java.nio.file.NoSuchFileException in search: Blob object not found: 404 Not Found #131346
Description
Elasticsearch Version
git-fbf9f4fc012a (serverless)
Installed Plugins
No response
Java Version
bundled
OS Version
Ubuntu 20.04
Problem Description
Got this suppressed error:
org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
at [email protected]/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:237)
at [email protected]/org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:143)
at [email protected]/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:70)
at [email protected]/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$39(IndicesService.java:1681)
at [email protected]/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$40(IndicesService.java:1747)
at [email protected]/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:165)
at [email protected]/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:148)
at [email protected]/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:440)
at [email protected]/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:111)
at [email protected]/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1753)
at [email protected]/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1675)
at [email protected]/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:670)
at [email protected]/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:877)
at [email protected]/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:710)
at [email protected]/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
at [email protected]/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
at [email protected]/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
at [email protected]/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
at java.base/java.lang.Thread.run(Thread.java:1447)
Caused by: java.io.IOException: java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: Blob object [indices/Or_RZzu8RuGWZGqLpYICpw/0/1/stateless_commit_4415] not found: 404 Not Found
GET https://storage.googleapis.com/download/storage/v1/b/hqc7io7q-production-elastic-k8s/o/indices%2FOr_RZzu8RuGWZGqLpYICpw%2F0%2F1%2Fstateless_commit_4415?alt=media
No such object: hqc7io7q-production-elastic-k8s/indices/Or_RZzu8RuGWZGqLpYICpw/0/1/stateless_commit_4415
at [email protected]/co.elastic.elasticsearch.stateless.lucene.BlobCacheIndexInput.readInternal(BlobCacheIndexInput.java:142)
at [email protected]/org.elasticsearch.blobcache.common.BlobCacheBufferedIndexInput.refill(BlobCacheBufferedIndexInput.java:304)
at [email protected]/org.elasticsearch.blobcache.common.BlobCacheBufferedIndexInput.readByte(BlobCacheBufferedIndexInput.java:57)
at [email protected]/org.apache.lucene.store.DataInput.readVInt(DataInput.java:121)
at [email protected]/org.elasticsearch.blobcache.common.BlobCacheBufferedIndexInput.readVInt(BlobCacheBufferedIndexInput.java:172)
at [email protected]/org.apache.lucene.util.bkd.BKDReader$BKDPointTree.readDocIDs(BKDReader.java:662)
at [email protected]/org.apache.lucene.util.bkd.BKDReader$BKDPointTree.visitDocValues(BKDReader.java:634)
at [email protected]/org.apache.lucene.util.bkd.BKDReader$BKDPointTree.visitLeavesOneByOne(BKDReader.java:621)
at [email protected]/org.apache.lucene.util.bkd.BKDReader$BKDPointTree.visitDocValues(BKDReader.java:615)
at [email protected]/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitablePointTree.visitDocValues(ExitableDirectoryReader.java:491)
at [email protected]/org.apache.lucene.search.PointRangeQuery$1.pointCount(PointRangeQuery.java:484)
at [email protected]/org.apache.lucene.search.PointRangeQuery$1.pointCount(PointRangeQuery.java:479)
at [email protected]/org.apache.lucene.search.PointRangeQuery$1.pointCount(PointRangeQuery.java:479)
at [email protected]/org.apache.lucene.search.PointRangeQuery$1.pointCount(PointRangeQuery.java:479)
at [email protected]/org.apache.lucene.search.PointRangeQuery$1.pointCount(PointRangeQuery.java:479)
at [email protected]/org.apache.lucene.search.PointRangeQuery$1.pointCount(PointRangeQuery.java:455)
at [email protected]/org.apache.lucene.search.PointRangeQuery$1.count(PointRangeQuery.java:405)
at [email protected]/org.apache.lucene.search.IndexOrDocValuesQuery$1.count(IndexOrDocValuesQuery.java:153)
at [email protected]/org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.count(LRUQueryCache.java:851)
at [email protected]/org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.count(IndicesQueryCache.java:171)
at [email protected]/org.apache.lucene.search.TotalHitCountCollector.getLeafCollector(TotalHitCountCollector.java:48)
at [email protected]/org.elasticsearch.search.query.PartialHitCountCollector.getLeafCollector(PartialHitCountCollector.java:47)
at [email protected]/org.elasticsearch.search.query.QueryPhaseCollector.getLeafCollector(QueryPhaseCollector.java:147)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:444)
at [email protected]/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:387)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:365)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
at [email protected]/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
at [email protected]/org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:111)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:369)
at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:336)
at [email protected]/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:212)
... 23 more
Caused by: java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: Blob object [indices/Or_RZzu8RuGWZGqLpYICpw/0/1/stateless_commit_4415] not found: 404 Not Found
GET https://storage.googleapis.com/download/storage/v1/b/hqc7io7q-production-elastic-k8s/o/indices%2FOr_RZzu8RuGWZGqLpYICpw%2F0%2F1%2Fstateless_commit_4415?alt=media
No such object: hqc7io7q-production-elastic-k8s/indices/Or_RZzu8RuGWZGqLpYICpw/0/1/stateless_commit_4415
at [email protected]/org.elasticsearch.action.support.PlainActionFuture$Sync.getValue(PlainActionFuture.java:279)
at [email protected]/org.elasticsearch.action.support.PlainActionFuture$Sync.get(PlainActionFuture.java:266)
at [email protected]/org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:96)
at [email protected]/org.elasticsearch.blobcache.shared.SharedBlobCacheService$CacheFile.readSingleRegion(SharedBlobCacheService.java:1310)
at [email protected]/org.elasticsearch.blobcache.shared.SharedBlobCacheService$CacheFile.populateAndRead(SharedBlobCacheService.java:1287)
at [email protected]/co.elastic.elasticsearch.stateless.cache.reader.CacheFileReader.doRead(CacheFileReader.java:124)
at [email protected]/co.elastic.elasticsearch.stateless.cache.reader.CacheFileReader.read(CacheFileReader.java:94)
at [email protected]/co.elastic.elasticsearch.stateless.lucene.BlobCacheIndexInput.readInternalSlow(BlobCacheIndexInput.java:175)
at [email protected]/co.elastic.elasticsearch.stateless.lucene.BlobCacheIndexInput.doReadInternal(BlobCacheIndexInput.java:167)
at [email protected]/co.elastic.elasticsearch.stateless.lucene.BlobCacheIndexInput.readInternal(BlobCacheIndexInput.java:131)
... 55 more
Caused by: java.nio.file.NoSuchFileException: Blob object [indices/Or_RZzu8RuGWZGqLpYICpw/0/1/stateless_commit_4415] not found: 404 Not Found
GET https://storage.googleapis.com/download/storage/v1/b/hqc7io7q-production-elastic-k8s/o/indices%2FOr_RZzu8RuGWZGqLpYICpw%2F0%2F1%2Fstateless_commit_4415?alt=media
No such object: hqc7io7q-production-elastic-k8s/indices/Or_RZzu8RuGWZGqLpYICpw/0/1/stateless_commit_4415
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRetryingInputStream.openStream(GoogleCloudStorageRetryingInputStream.java:131)
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRetryingInputStream.<init>(GoogleCloudStorageRetryingInputStream.java:77)
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:253)
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:73)
at [email protected]/co.elastic.elasticsearch.stateless.cache.reader.ObjectStoreCacheBlobReader.getRangeInputStream(ObjectStoreCacheBlobReader.java:58)
at [email protected]/co.elastic.elasticsearch.stateless.cache.reader.ObjectStoreCacheBlobReader.lambda$getRangeInputStream$0(ObjectStoreCacheBlobReader.java:67)
at [email protected]/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:58)
at [email protected]/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:55)
... 8 more
Steps to Reproduce
Comes from serverless alerts.
Logs (if relevant)
No response