UnsatisfiedLinkError: libopensearchknn_faiss_avx512_spr.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found

[naveentatikonda]

Description

Running into UnsatisfiedLinkError during runtime while loading libopensearchknn_faiss_avx512_spr.so library when tested with 2.19 RC2 docker setup for k-NN (which is using GCC 12.4). But, no issues when tested with tar artifact.

java.lang.UnsatisfiedLinkError: /usr/share/opensearch/plugins/opensearch-knn/lib/libopensearchknn_faiss_avx512_spr.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /usr/share/opensearch/plugins/opensearch-knn/lib/libopensearchknn_faiss_avx512_spr.so)
        at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
        at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331)
        at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197)
        at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139)
        at java.base/jdk.internal.loader.NativeLibraries.findFromPaths(NativeLibraries.java:259)
        at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:251)
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2451)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:916)
        at java.base/java.lang.System.loadLibrary(System.java:2063)
        at org.opensearch.knn.jni.FaissService.lambda$static$0(FaissService.java:48)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
        at org.opensearch.knn.jni.FaissService.<clinit>(FaissService.java:42)
        at org.opensearch.knn.jni.JNIService.initIndex(JNIService.java:54)
        at org.opensearch.knn.index.codec.nativeindex.MemOptimizedNativeIndexBuildStrategy.lambda$buildAndWriteIndex$0(MemOptimizedNativeIndexBuildStrategy.java:63)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
        at org.opensearch.knn.index.codec.nativeindex.MemOptimizedNativeIndexBuildStrategy.buildAndWriteIndex(MemOptimizedNativeIndexBuildStrategy.java:62)
        at org.opensearch.knn.index.codec.nativeindex.NativeIndexWriter.buildAndWriteIndex(NativeIndexWriter.java:149)
        at org.opensearch.knn.index.codec.nativeindex.NativeIndexWriter.flushIndex(NativeIndexWriter.java:104)
        at org.opensearch.knn.index.codec.KNN990Codec.NativeEngines990KnnVectorsWriter.flush(NativeEngines990KnnVectorsWriter.java:121)
        at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.flush(PerFieldKnnVectorsFormat.java:115)
        at org.apache.lucene.index.VectorValuesConsumer.flush(VectorValuesConsumer.java:76)
        at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:296)
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:445)
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
        at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:579)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355)
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345)
        at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
        at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
        at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72)
        at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52)
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
        at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
        at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:434)
        at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:414)
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
        at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:213)
        at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1890)
        at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1869)
        at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4705)
        at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1300)
        at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1444)
        at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:955)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

[naveentatikonda]

@peterzhuamazon can you pls take a look into this issue. Thanks!

[peterzhuamazon]

AL2023, which is our release image base, have lrwxrwxrwx 1 root root 19 Sep 5 2023 /lib64/libstdc++.so.6 -> libstdc++.so.6.0.29 by default.

AL2, which is our build image base, initially have lrwxrwxrwx 1 root root 19 Jan 8 06:16 /lib64/libstdc++.so.6 -> libstdc++.so.6.0.24 which is within the compatible range of AL2023.

However, since opensearch-project/opensearch-build#5226 AL2 for X64 has upgraded gcc from 10 to 12, which subsequently upgrade libstdc++.so to lrwxrwxrwx 1 root root 19 Jan 25 01:42 /lib64/libstdc++.so.6 -> libstdc++.so.6.0.30 which caused the error.

[peterzhuamazon]

One solution is to also upgrade the gcc version on AL2023 so that the libgcc version is increased as well on x64.

[peterzhuamazon]

Hi @naveentatikonda could you provide a few API calls for me to test the change?

Thanks.

[naveentatikonda]

@peterzhuamazon here are some sample curl commands to test

// Create Index

curl -X PUT "localhost:9200/test-index?pretty" -H 'Content-Type: application/json' -d'
{
    "settings": {
        "index": {
            "knn": true,
            "knn.algo_param.ef_search": 100,
            "knn.advanced.approximate_threshold": 0
        }
    },
    "mappings": {
        "properties": {
            "my_vector1": {
                "type": "knn_vector",
                "dimension": 16,
                "method": {
                    "name": "hnsw",
                    "space_type": "l2",
                    "engine": "faiss",
                    "parameters": {
                        "encoder": {
                            "name": "sq",
                            "parameters": {
                            "type": "fp16"
                             }
                        }
                    }
                }
            }
        }
    }
}
'

// Ingest docs

curl -X POST "localhost:9200/_bulk?pretty" -H 'Content-Type: application/json' -d'
{ "index": { "_index": "test-index", "_id": "1" } }
{ "my_vector1": [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160]}
{ "index": { "_index": "test-index", "_id": "2" } }
{ "my_vector1": [11, 21, 130, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 260]}
'

// Query index

curl -X GET "localhost:9200/test-index/_search?pretty" -H 'Content-Type: application/json' -d'
{ "size": 2,
 "query": {
   "knn": {
     "my_vector1": {
       "vector": [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160],
       "k": 2
     }
   }
 }
}
'

[peterzhuamazon]

Able to achieve the output here, will PR:

{
  "took" : 29,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test-index",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "my_vector1" : [
            10,
            20,
            30,
            40,
            50,
            60,
            70,
            80,
            90,
            100,
            110,
            120,
            130,
            140,
            150,
            160
          ]
        }
      },
      {
        "_index" : "test-index",
        "_id" : "2",
        "_score" : 4.99925E-5,
        "_source" : {
          "my_vector1" : [
            11,
            21,
            130,
            40,
            50,
            60,
            70,
            80,
            90,
            100,
            110,
            120,
            130,
            140,
            150,
            260
          ]
        }
      }
    ]
  }
}

[peterzhuamazon]

• Update OS release docker with libstdc++.so.6.0.30 for k-NN avx512_spr opensearch-build#5291
• Ensure libstdc++.so.6.0.30 only replace on x64 docker image opensearch-build#5294
• Ensure AL2 build image keep libstdc++.so.6.0.24 version opensearch-build#5306
• Update AL2 x64 build image to have both gcc10 and gcc12 opensearch-build#5311
• Update build.gradle and build.sh to separate x64 linux nmslib build with different gcc versions #2506

[peterzhuamazon]

We are changing the strategy on this one.

After building the gcc12 we will replace the newly released 6.0.30 back to 6.0.24 version of libstdgc++ to avoid using newer version for better backward compatibility due to bugs on barebone AL2 and AL2023 not compatible with 6.0.30.

fatal error in thread [opensearch[data1][refresh][T#2]], exiting
java.lang.UnsatisfiedLinkError: /home/ec2-user/os2.19/rc4/opensearch-2.19.0/plugins/opensearch-knn/lib/libopensearchknn_faiss_avx512.so: /home/ec2-user/os2.19/rc4/opensearch-2.19.0/data/ml_cache/pytorch/1.13.1-cpu-precxx11-linux-x86_64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/ec2-user/os2.19/rc4/opensearch-2.19.0/plugins/opensearch-knn/lib/libopensearchknn_faiss_avx512.so)
        at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
        at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331)
        at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197)
        at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139)
        at java.base/jdk.internal.loader.NativeLibraries.findFromPaths(NativeLibraries.java:259)
        at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:251)
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2451)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:916)
        at java.base/java.lang.System.loadLibrary(System.java:2063)
        at org.opensearch.knn.jni.FaissService.lambda$static$0(FaissService.java:50)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
        at org.opensearch.knn.jni.FaissService.<clinit>(FaissService.java:42)
        at org.opensearch.knn.jni.JNIService.initIndex(JNIService.java:54)
        at org.opensearch.knn.index.codec.nativeindex.MemOptimizedNativeIndexBuildStrategy.lambda$buildAndWriteIndex$0(MemOptimizedNativeIndexBuildStrategy.java:63)

Thanks.

[peterzhuamazon]

More errors, this time in nmslib:

/opt/java/openjdk-21/bin/java: symbol lookup error: /tmp/tmpw0cjldal/1/local-test-cluster/opensearch-2.19.0/plugins/opensearch-knn/lib/libopensearchknn_nmslib.so: undefined symbol: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev

[peterzhuamazon]

The issue is related to nmslib do not support gcc11+, while faiss needs gcc12+ to build. cc: @naveentatikonda
This means we need to install gcc10 for AL2 x64 build image, but not replace existing gcc12 version.

Then update build.gradle of k-NN repo to dynamically switch GCC with env vars. Thanks.