jruby-9.0.5.0-complete.jar: leaves stale jffi*.dll files in temp folder

[perlun]

This seems very similar to #1237, which was resolved on the JRuby 1.7 track a while ago.

Symptoms: on a customer Windows server, the C: folder was filled up and the application(s) stopped working correctly. Upon careful examination, we discovered loads of temp files.

The jruby-* subfolders have content that looks like this:

(I treat this as two different issues, just leaving the images here for the sake of completeness. Will follow this up a bit more myself but if anyone has any obvious ideas, please let me know.)

---

The jffi files turned out to be very simple to reproduce locally:

$ java -jar jruby-complete-9.0.5.0.jar -e "require 'ffi'"

This will create a jffi file in the temp folder (in my case, C:\msys64\tmp).

I also tried downloading jruby-bin-9.0.5.0.zip and running with that. Surprisingly enough, the result is different:

plundberg@DESKTOP-NVMNDAM MSYS /c/jruby-9.0.5.0
$ bin/jruby -v
jruby 9.0.5.0 (2.2.3) 2016-01-26 7bee00d Java HotSpot(TM) 64-Bit Server VM 25.66-b18 on 1.8.0_66-b18 +jit [Windows 10-amd64]

plundberg@DESKTOP-NVMNDAM MSYS /c/jruby-9.0.5.0
$ bin/jruby -e "require 'ffi'"

This does not create any stale temp files whatsoever. 😲

Any clues?

[perlun]

(I looked at the other problem, with the stale .jar files left in a temp folder. It was very similar in nature: require 'openssl' and you get it. Same behavior there; it appears only with jruby-complete, not with the "normal" jruby.)

[mkristian]

the original problem you are referring to was when you kill the jruby
process hard, then the JVM can not clean up files marked as
deleteOnExit() since the process is gone. this is true only for
jruby-complete or any scenario where you need to have native libraries or
jars loaded from within a jar-file (in your case the jruby-complete.jar)
which are copied to an temp file and used from there. you are saying that
on windows those file stay around, i.e. the JVM does not delete them on
exit.

[perlun]

Correct. But this time, it's not only when you kill the jruby process hard. It happens all the time, every time. It's totally deterministic. So, maybe this is an entirely different source of the problem than in the original issue.

Any ideas on where we can start digging in this? I just downloaded jruby and set up the local build environment. Verified that it's still a problem with latest HEAD.

If someone else wants to take over the debugging, feel free. 😄

[mkristian]

@perlun that are the two places I know where temp files are created:
https://github.com/jruby/jruby/blob/master/core/src/main/java/org/jruby/util/JRubyClassLoader.java#L87
https://github.com/jnr/jffi/blob/master/src/main/java/com/kenai/jffi/internal/StubLoader.java#L344

[rsim]

I see the same issue on JRuby 1.7.24 that after exiting JRuby it leaves jffi*.dll file in temp directory (which is specified in java.io.tmpdir Java property).

In addition, I see jruby-* directories created with jruby*openssl.jar, jruby*bcpkix*.jar, and jruby*bcprov*.jar files created. These jruby-* directories in temp directory remain as well after exiting JRuby.

[kares]

win won't delete any files that are "still" open ... which might just fit this case esp. the StubLoader's.
the native library gets loaded and it won't get unloaded until its finalizer is run (which is likely only around the class loader gets released) ... might just happen after the JVM cleans up the deleteOnExist temp files.

[perlun]

I think we should modify the code to clean up the tempfiles using a JVM shutdown hook instead: https://dzone.com/articles/know-jvm-series-2-shutdown

[kares]

@perlun could you test it actually helps the windows case esp. when a temp file is a loaded .dll library ?
... cause deleteOnExit already does File#delete from a shutdown hook but maybe the order will matter.

[mkristian]

we have/had a similar problem with a I think it is redstorm gem which spawns java/jruby processes and when they are not needed anymore they are kill hard, i.e. the JVM has no chance to cleanup anything. this resulted in filling up the /tmp directory (under linux) and eventually running out of disk space within a few days :(

the solution or workaround this problem is cleanup the /tmp directory on each new startup of jruby. the lookout-jruby is doing this: https://github.com/lookout/lookout-jruby/blob/master/lib/lookout/jruby.rb#L10

reading all this here, I think that could be at least something JRuby itself could do: cleanup stale temp files on startup. of course there is always a set of native dll and/or jars laying around but pollution of the temp space is limited.