Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sbt-assembly is using lots of native memory #517

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

ashangit
Copy link

Issue description

On some of our build relying on sbt-assembly to create shaded jar we have observed big usage of native memory.
For a 6G max heap JVM, the pod memory usage is above 12G

Analysing the native usage with libtcmalloc_and_profiler.so.4 and google-pprof we observed that the native usage is due to native usage of Inflater used to unzip jar:
malloc

Src code ref of the native inflater code: Java_java_util_zip_Inflater_init and Java_java_util_zip_Inflater_inflateBytesBytes

Proposal

Current code keep all jar file handler which leads to all native inflater being kept with all the memory those are using

In order to avoid this, the proposal is to load all entry data from each jar as ByteArrayInputStream instead of keeping the stream and immediately close the jar once all entries are loaded

Here the native memory usage after this change
malloc_after

The counterpart is an increase of the heap usage.
On our specific build we had to increase heap size to 7G but pod memory usage has decreased below 10G:
Screenshot 2024-01-18 at 15 25 29

.entries()
.asScala
.filterNot(_.isDirectory)
.toVector
.par
.flatMap { entry =>
jarShader(module)(entry.getName, () => jarFile.getInputStream(entry))
jarShader(module)(entry.getName, () => new ByteArrayInputStream(jarFile.getInputStream(entry).readAllBytes()))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this is a good idea. Maybe it would be better to create a setting to tweak the parallelism of .par to reduce the concurrent calls on very big JARs?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updating parallelism of .par should not change the native memory usage as we will still keep all those jar handler

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you're right. The PR as it stands requires heap equivalent to all JAR file size right?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No it will store the inflated data for each entries of each jar. From heapdump on our build job it represent around 5GB of byte[] while using the "stream" version it use around 3.5GB of byte[]. Stream version also used lots of byte[] because even if it is a stream the stream as already load/initiate a chunk of bytes per entry which seems to represent a big percent of an entry (probably because each of them are mostly only little text file).
Another approach would be to not load the stream in https://github.com/sbt/sbt-assembly/blob/develop/src/main/scala/sbtassembly/Assembly.scala#L304. Just get needed information for each entry and close the jar file handler immediately
And load the stream only in createJar.
This would reduce the heap used (even with current release) and reduce off heap size
Do you think it is possible or there are some other usage of the stream that would be uneasy to extract?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jar Jar Abrams needs to read the streams to process various rules against the stream of bytecode I think. Similar to what it's doing () => jarFile.getInputStream(entry) a lazy function that creates a stream on demand, I guess we can further make it lazier to provide:

name => {
  val jarFile = lookupJarFile(jar.data)
  val entry = jarFile.getEntry(name)
  jarFile.getInputStream(entry)
}

where we can keep a LRU cache of JAR files in lookupJarFile(...)?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shader that rely on this jarjarabrams Shader seems to already load the whole stream into a ByteArrayInputStream
From jarjar-abrams bytecodeShader signature: https://github.com/eed3si9n/jarjar-abrams/blob/develop/core/src/main/scala/com/eed3si9n/jarjarabrams/Shader.scala#L71

I'm not sure to see the gain in keeping the stream as we will load all the bytes in the shader. Also wondering if we rely on such mechanism with LRU cache if we could not reach some bugs with LRU cache closing the file handler while still having an input stream open from it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants