Enable std-database in native image (#12068)

* Enable std-database in native image

Not working ATM, breaks even BASE stdlib tests.
Culprit appears to be `sqlite-jdbc` client

* Workaround for sqlite-jdbc NI properies bug

sqlite-jdbc's NI properties file defines
`--enable-url-protocols=jar`. That's experimental and breaks completely
our `HostClassloader`. We exclude their config but include their feature
to correctly extract shared native library.

* fmt

* Towards error-free Table_Tests

* More reflect updates to make NI happy

* Segfaults in NI

* Pass all Table_Tests except for Excel_Spec

Excel_Spec leads to segfault, potentially related to reading/writing
files.

* Make sure `awt` is included in NI resource configs

Apparently `poi` requires `java.awt` and it was segfaulting in a
mysterious way when it was missing.

* fix moduledeps inconsistency

* Re-shufflings, de-duplication in resource configs

* legal

* Remove xalan bundle

* PR review

* nit

* Update table's reflect config

CI fails with a useless `java.lang.Exception` which is not reproducible
locally. Attempting to blindly add a few methods/fields, hoping that it
will fix it.

* Potentially fix Windows NI failures

* Workaround java.awt dependency via substitution

java.awt support on MacOS is still missing therefore
bundling/configuring resources for it only makes sense on Linux/Windows.
That was unsatisafactory.

This change uses native image's substitution mechanism to avoid any awt
library loading. The latter will not be functional when one attempts to
use it but our implementation does not rely on it at all.

* fmt

build.sbt

@@ -1920,7 +1920,8 @@ lazy val `ydoc-polyfill` = project
       "com.github.sbt"       % "junit-interface"             % junitIfVersion            % Test
     ),
     libraryDependencies ++= {
-      GraalVM.modules ++ GraalVM.jsPkgs ++ GraalVM.chromeInspectorPkgs ++ helidon
+      GraalVM.modules ++ GraalVM.jsPkgs
+        .map(_ % "provided") ++ GraalVM.chromeInspectorPkgs ++ helidon
     }
   )
   .dependsOn(`syntax-rust-definition`)
@@ -3717,7 +3718,8 @@ lazy val `engine-runner` = project
       val stdLibsJars =
         `base-polyglot-root`.listFiles("*.jar").map(_.getAbsolutePath()) ++
         `image-polyglot-root`.listFiles("*.jar").map(_.getAbsolutePath()) ++
-        `table-polyglot-root`.listFiles("*.jar").map(_.getAbsolutePath())
+        `table-polyglot-root`.listFiles("*.jar").map(_.getAbsolutePath()) ++
+        `database-polyglot-root`.listFiles("*.jar").map(_.getAbsolutePath())
       core ++ stdLibsJars
     },
     buildSmallJdk := {
@@ -3728,7 +3730,7 @@ lazy val `engine-runner` = project
       val NI_MODULES =
         "org.graalvm.nativeimage,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.base,org.graalvm.nativeimage.driver,org.graalvm.nativeimage.librarysupport,org.graalvm.nativeimage.objectfile,org.graalvm.nativeimage.pointsto,com.oracle.graal.graal_enterprise,com.oracle.svm.svm_enterprise"
       val JDK_MODULES =
-        "jdk.localedata,jdk.httpserver,java.naming,java.net.http"
+        "jdk.localedata,jdk.httpserver,java.naming,java.net.http,java.desktop"
       val DEBUG_MODULES  = "jdk.jdwp.agent"
       val PYTHON_MODULES = "jdk.security.auth,java.naming"
 
@@ -3791,6 +3793,12 @@ lazy val `engine-runner` = project
             "enso",
             targetDir     = engineDistributionRoot.value / "bin",
             staticOnLinux = false,
+            // sqlite-jdbc includes `--enable-url-protocols=jar` in its native-image.properites file,
+            // which breaks all our class loading. We still want to run `SqliteJdbcFeature` which extracts a proper
+            // native library from the jar.
+            excludeConfigs = Seq(
+              s".*sqlite-jdbc-.*\\.jar,META-INF/native-image/org\\.xerial/sqlite-jdbc/native-image\\.properties"
+            ),
             additionalOptions = Seq(
               "-Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.NoOpLog",
               "-H:IncludeResources=.*Main.enso$",
@@ -3802,8 +3810,11 @@ lazy val `engine-runner` = project
               // "-H:-DeleteLocalSymbols",
               // you may need to set smallJdk := None to use following flags:
               // "--trace-class-initialization=org.enso.syntax2.Parser",
+              // "--diagnostics-mode",
+              // "--verbose",
               "-Dnic=nic",
-              "-Dorg.enso.feature.native.lib.output=" + (engineDistributionRoot.value / "bin")
+              "-Dorg.enso.feature.native.lib.output=" + (engineDistributionRoot.value / "bin"),
+              "-Dorg.sqlite.lib.exportPath=" + (engineDistributionRoot.value / "bin")
             ),
             mainClass = Some("org.enso.runner.Main"),
             initializeAtRuntime = Seq(
@@ -3825,6 +3836,7 @@ lazy val `engine-runner` = project
               "org.enso.base",
               "org.enso.image",
               "org.enso.table",
+              "org.enso.database",
               "org.eclipse.jgit"
             )
           )
@@ -4734,7 +4746,8 @@ lazy val `std-table` = project
       "org.apache.poi"           % "poi-ooxml"               % poiOoxmlVersion,
       "org.apache.xmlbeans"      % "xmlbeans"                % xmlbeansVersion,
       "org.antlr"                % "antlr4-runtime"          % antlrVersion,
-      "org.apache.logging.log4j" % "log4j-to-slf4j"          % "2.18.0" // org.apache.poi uses log4j
+      "org.apache.logging.log4j" % "log4j"                   % "2.24.3",
+      "org.apache.logging.log4j" % "log4j-to-slf4j"          % "2.24.3" // org.apache.poi uses log4j
     ),
     Compile / packageBin := Def.task {
       val result = (Compile / packageBin).value

build_tools/build/src/engine/context.rs

@@ -696,6 +696,12 @@ pub async fn runner_sanity_test(
             .run_ok()
             .await;
 
+        let test_table = Command::new(&enso)
+            .args(["--run", repo_root.test.join("Table_Tests").as_str()])
+            .set_env(ENSO_DATA_DIRECTORY, engine_package)?
+            .run_ok()
+            .await;
+
         let test_geo = Command::new(&enso)
             .args(["--run", repo_root.test.join("Geo_Tests").as_str()])
             .set_env(ENSO_DATA_DIRECTORY, engine_package)?
@@ -708,7 +714,7 @@ pub async fn runner_sanity_test(
             .run_ok()
             .await;
 
-        let all_cmds = test_base.and(test_internal_base).and(test_geo).and(test_image);
+        let all_cmds = test_base.and(test_internal_base).and(test_table).and(test_geo).and(test_image);
 
         // The following test does not actually run anything, it just checks if the engine
         // can accept `--jvm` argument and evaluates something.

distribution/engine/THIRD-PARTY/NOTICE

@@ -563,7 +563,7 @@ Copyright notices related to this dependency can be found in the directory `org.
 
 'slf4j-api', licensed under the MIT License, is distributed with the engine.
 The license file can be found at `licenses/MIT`.
-Copyright notices related to this dependency can be found in the directory `org.slf4j.slf4j-api-2.0.9`.
+Copyright notices related to this dependency can be found in the directory `org.slf4j.slf4j-api-2.0.16`.
 
 
 'cats-core_2.13', licensed under the MIT, is distributed with the engine.

distribution/engine/THIRD-PARTY/org.slf4j.slf4j-api-2.0.16/NOTICES

@@ -0,0 +1,32 @@
+/**
+ * Copyright (c) 2004-2011 QOS.ch
+ * All rights reserved.
+ *
+ * Permission is hereby granted, free  of charge, to any person obtaining
+ * a  copy  of this  software  and  associated  documentation files  (the
+ * "Software"), to  deal in  the Software without  restriction, including
+ * without limitation  the rights to  use, copy, modify,  merge, publish,
+ * distribute,  sublicense, and/or sell  copies of  the Software,  and to
+ * permit persons to whom the Software  is furnished to do so, subject to
+ * the following conditions:
+ *
+ * The  above  copyright  notice  and  this permission  notice  shall  be
+ * included in all copies or substantial portions of the Software.
+ *
+ * THE  SOFTWARE IS  PROVIDED  "AS  IS", WITHOUT  WARRANTY  OF ANY  KIND,
+ * EXPRESS OR  IMPLIED, INCLUDING  BUT NOT LIMITED  TO THE  WARRANTIES OF
+ * MERCHANTABILITY,    FITNESS    FOR    A   PARTICULAR    PURPOSE    AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+ * LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+ * OF CONTRACT, TORT OR OTHERWISE,  ARISING FROM, OUT OF OR IN CONNECTION
+ * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+Copyright (c) 2004-2019 QOS.ch
+
+Copyright (c) 2004-2021 QOS.ch
+
+Copyright (c) 2004-2022 QOS.ch
+
+Copyright (c) 2004-2024 QOS.ch

distribution/launcher/THIRD-PARTY/NOTICE

@@ -113,7 +113,7 @@ Copyright notices related to this dependency can be found in the directory `org.
 
 'slf4j-api', licensed under the MIT License, is distributed with the launcher.
 The license file can be found at `licenses/MIT`.
-Copyright notices related to this dependency can be found in the directory `org.slf4j.slf4j-api-2.0.9`.
+Copyright notices related to this dependency can be found in the directory `org.slf4j.slf4j-api-2.0.16`.
 
 
 'cats-core_2.13', licensed under the MIT, is distributed with the launcher.

distribution/engine/THIRD-PARTY/org.slf4j.slf4j-api-2.0.9/NOTICES → distribution/launcher/THIRD-PARTY/org.slf4j.slf4j-api-2.0.16/NOTICES

@@ -5,3 +5,5 @@ Copyright (c) 2004-2019 QOS.ch
 Copyright (c) 2004-2021 QOS.ch
 
 Copyright (c) 2004-2022 QOS.ch
+
+Copyright (c) 2004-2024 QOS.ch

distribution/lib/Standard/Table/0.0.0-dev/THIRD-PARTY/NOTICE

@@ -46,14 +46,19 @@ The license information can be found along with the copyright notices.
 Copyright notices related to this dependency can be found in the directory `org.apache.commons.commons-math3-3.6.1`.
 
 
-'log4j-api', licensed under the Apache License, Version 2.0, is distributed with the Table.
-The license information can be found along with the copyright notices.
-Copyright notices related to this dependency can be found in the directory `org.apache.logging.log4j.log4j-api-2.18.0`.
+'log4j', licensed under the Apache-2.0, is distributed with the Table.
+The license file can be found at `licenses/APACHE2.0`.
+Copyright notices related to this dependency can be found in the directory `org.apache.logging.log4j.log4j-2.24.3`.
+
+
+'log4j-api', licensed under the Apache-2.0, is distributed with the Table.
+The license file can be found at `licenses/APACHE2.0`.
+Copyright notices related to this dependency can be found in the directory `org.apache.logging.log4j.log4j-api-2.24.3`.
 
 
-'log4j-to-slf4j', licensed under the Apache License, Version 2.0, is distributed with the Table.
+'log4j-to-slf4j', licensed under the Apache-2.0, is distributed with the Table.
 The license file can be found at `licenses/APACHE2.0`.
-Copyright notices related to this dependency can be found in the directory `org.apache.logging.log4j.log4j-to-slf4j-2.18.0`.
+Copyright notices related to this dependency can be found in the directory `org.apache.logging.log4j.log4j-to-slf4j-2.24.3`.
 
 
 'poi', licensed under the Apache License, Version 2.0, is distributed with the Table.
@@ -78,5 +83,5 @@ Copyright notices related to this dependency can be found in the directory `org.
 
 'slf4j-api', licensed under the MIT License, is distributed with the Table.
 The license file can be found at `licenses/MIT`.
-Copyright notices related to this dependency can be found in the directory `org.slf4j.slf4j-api-1.7.36`.
+Copyright notices related to this dependency can be found in the directory `org.slf4j.slf4j-api-2.0.16`.
 

distribution/lib/Standard/Table/0.0.0-dev/THIRD-PARTY/org.apache.logging.log4j.log4j-2.24.3/NOTICES

@@ -0,0 +1,26 @@
+Copyright 2003-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (https://www.apache.org/).
+
+This product contains parts that were originally based on software from BEA.
+Copyright (c) 2000-2003, BEA Systems, <http://www.bea.com/> (dead link),
+which was acquired by Oracle Corporation in 2008.
+<http://www.oracle.com/us/corporate/Acquisitions/bea/index.html>
+<https://en.wikipedia.org/wiki/BEA_Systems>
+Note: The ASF Secretary has on hand a Software Grant Agreement (SGA) from
+BEA Systems, Inc. dated 9 Sep 2003 for XMLBeans signed by their EVP/CFO.
+
+This product contains W3C XML Schema documents. Copyright 2001-2003 (c)
+World Wide Web Consortium (Massachusetts Institute of Technology, European
+Research Consortium for Informatics and Mathematics, Keio University)
+
+This product contains the chunks_parse_cmds.tbl file from the vsdump program.
+Copyright (C) 2006-2007 Valek Filippov ([email protected])
+
+This product contains parts of the eID Applet project
+<http://eid-applet.googlecode.com> and <https://github.com/e-Contract/eid-applet>.
+Copyright (c) 2009-2018
+FedICT (federal ICT department of Belgium), e-Contract.be BVBA (https://www.e-contract.be),
+Bart Hanssens from FedICT
+