transport: error while dialing: dial unix /tmp/plugin1193340813: connect: no such file or directory

[jtabbert]

Describe the bug

transport: error while dialing: dial unix /tmp/plugin1193340813: connect: no such file or directory

In our environment, we have implemented JFrog Artifactory Secrets Engine v1.5.0

We have followed the usual approach to install plugin binary on the filesystem, register the plugin using Vault CLI, mount the engine and configure. Plugin was working well for some time.

After an unknown amount of time (3 weeks? 2 months?) we experienced issue interacting with the mount:

$ vault read artifactory/config/admin
"error":"1 error occurred:\n\t* rpc error: code = Unavailable desc = connection error: desc = \"transport: error while dialing: dial unix /tmp/plugin1193340813: connect: no such file or directory\"
Note that we implement PrivateTmp=yes in the systemd unit file and thus the file /tmp/plugin1193340813 does not actually exist, it would leverage the private filesystem in /tmp as created by systemd (eg. /tmp/systemd-private-f713ec85017f4d13aeda1aba94a901cd-vault.service-j8khAI/tmp) 

Restarting the Vault service resolved the issue

We have confirmed that there is no process which cleans the PrivateTmp dir, RHEL 8 OS specifically excludes PrivateTmp directories and no external process is cleaning this dir.

We were not able to identify any issue by review Vault operational or audit logs or OS logs (dmesg/journalctl)

To Reproduce
Steps to reproduce the behavior:

This has been a challenging error to reproduce. This error has occurred in 2 dev environments, and also UAT environment.

The internal investigation has shown that this error is specific to the JFrog Artifactory secrets engine. The Vault Oracle custom plugin is not experiencing this error and has been operating normally.

It's our understanding that if the /tmp directory contents are deleted it would cause this error, but as stated above this appears to not be the case as the Oracle plugin is working without issue.

This issue is preventing the secrets engine from being used in production

We will continue to determine steps to reproduce.

Requirements for and issue

• A description of the bug
• [n/a] A fully functioning vault configuration snippet that can be copy&pasted (no outside files or ENV vars unless that's part of the issue). If this is not supplied, this issue will likely be closed without any effort expended.
• [7.63.22] Your version of artifactory (you can curl it at $host/artifactory/api/system/version
• [1.15.6] Your version of vault
• [1.5.0] Your version of vault plugin

Expected behavior

Vault Artifactory plugin continues to operate & communicate with Vault process

Screenshots
n/a

Desktop (please complete the following information):
n/a

Smartphone (please complete the following information):
n/a

Additional context
Add any other context about the problem here.

[alexhung]

@jtabbert Thanks for the report.

The Artifactory plugin code doesn't use the tmp directory at all.

My guess is that Oracle plugin is a database plugin whereas Artifactory one is a secrets plugin, and perhaps Vault treats them differently.

Have you tried setting a different tmp dir using the plugin_tmpdir configuration? https://developer.hashicorp.com/vault/docs/configuration#plugin_tmpdir

[jtabbert]

Hi Alex, Thanks for your response

RE: plugin_tmpdir

Hashi documentation for this setting indicate “If not set, Vault will use the system's default directory for temporary files. Generally not necessary unless you are using containerized plugins and Vault does not share a temporary folder with other processes, such as if using systemd's PrivateTmp setting.” In our case we are using PrivateTmp=yes in systemd unit file. We have not attempted to use another tmp dir using this configuration setting

RE: tmp directory is not used by Artifactory plugin code

Using lsof we can see that vault process has created the /tmp/plugin2262638940 file in systemd private tmp and not the plugin binary itself. We have engaged HashiCorp Support to further discuss regarding plugin architecture and UNIX socket files for inter-process communications. Are you able to shed any light here from your experience? I can imagine in your environment this tmp file is created similarly

# lsof +E | grep plugin2262638940

COMMAND       PID     TID TASKCMD                      USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME

vault     1070534                                     vault   25u     unix 0xffff94adab2acd80       0t0   56081022 /tmp/plugin2262638940 type=STREAM

However, please note that lsof for the artifactory-secrets-plugin process did indicate an open tmp file, Vault Operational logs also indicate this tmp file was created, however this file could not be located on the system. So I am unclear about the statement that plugin does not use tmp directory at all

# lsof +E | grep 1070680 

…

artifacto 1070680 1070687 artifacto                   vault   14u     unix 0xffff94adab2adf80       0t0   56081020 /tmp/plugin1072889969 type=STREAM ->INO=56081019 1070534,vault,23u

 

# lsof +E | grep 56081019

…

vault     1070534                                     vault   23u     unix 0xffff94adab2adb00       0t0   56081019 type=STREAM ->INO=56081020 1070680,artifacto,14u

…

artifacto 1070680                                     vault   14u     unix 0xffff94adab2adf80       0t0   56081020 /tmp/plugin1072889969 type=STREAM ->INO=56081019 1070534,vault,23u

 

# grep plugin1072889969 vault.log

Oct 16 13:31:52 hvault05-dev vault[1070534]: 2024-10-16T13:31:52.305-0400 [DEBUG] secrets.artifactory.artifactory_ab289714.artifactory.artifactory-secrets-plugin: plugin address: address=/tmp/plugin1072889969 network=unix timestamp=2024-10-16T13:31:52.305-0400

 

# find / -name plugin1072889969

#

Thanks and regards,

[alexhung]

So I am unclear about the statement that plugin does not use tmp directory at all

@jtabbert By that I mean the code isn't directly using the temp directory for files, streams, etc. It is possible that the Vault SDK utilizes the temp directory when the plugin uses its functionality. My guess would be something like logging which needs to be sent back to the Vault process in order to be consolidated with other Vault and plugin logs.

I'm interested in Vault response, so if you can please keep me informed. Thanks!