out_file: add symlink_path_use_relative option to use relative path instead of absolute path in symlink_path

[domna]

Which issue(s) this PR fixes:
None.

What this PR does / why we need it:
This PR moves from an absolute link to a relative link for symlink_path parameter. Otherwise, this does not work in a dockerized environment where the path is wired to the the outside of the container.
I can also add an additional parameter to switch between absolute and relative linking if necessary and/or desired.

Docs Changes:
TODO https://docs.fluentd.org/input/tail

Release Note:
Same as the title.

[Watson1978]

It might be better to keep the default behavior.
If you want to use relative paths, please use the option function.

[daipom]

Thanks for this PR!

This PR moves from an absolute link to a relative link for symlink_path parameter. Otherwise, this does not work in a dockerized environment where the path is wired to the the outside of the container.

Could you please explain in more detail?
How do you use this relative simlink?

[domna]

Could you please explain in more detail?
How do you use this relative simlink?

Sure. In my project I'm using fluentd to collect the logs in a docker compose stack. For some containers I want to write file based logs to disk, i.e., I mount them from the external fs into the container. I use the symlink_path to simulate some sort of log rotate pattern (where I have one fixed file keeping the latest logs), which I need for the application running on the bare metal system.
Now, using the symlink_path (as implemented before) works inside the fluentd container but it breaks on the outside, because the absolute path of the directory is obviously different.
I hope this makes it a bit clearer...

It might be better to keep the default behavior.
If you want to use relative paths, please use the option function.

You mean keeping the behaviour of symlink_path as an absolute link and adding an additional option, e.g., relative_symlink_path = true/false to switch to relative links? I can certainly implement it like this.

[daipom]

Thanks! Please let me confirm a few more things.

Sure. In my project I'm using fluentd to collect the logs in a docker compose stack. For some containers I want to write file based logs to disk, i.e., I mount them from the external fs into the container.

So Fluentd collects logs in a container, but those logs are actually in an external file system, and it is mounted to the container.

I use the symlink_path to simulate some sort of log rotate pattern (where I have one fixed file keeping the latest logs), which I need for the application running on the bare metal system.

Does this mean that the external application outside the container collects logs from the symlink inside the container?

Now, using the symlink_path (as implemented before) works inside the fluentd container but it breaks on the outside, because the absolute path of the directory is obviously different.

This is still not clear to me.
If the symlink is a relative path, how does an external application read it?

[Watson1978]

Can't you use Docker Logging Driver in your use case?

[domna]

So Fluentd collects logs in a container, but those logs are actually in an external file system, and it is mounted to the container.

On the system fluentd collects logs from all running containers through the docker logging driver. I directly forward all logs to a loki storage but for two services I additionally create files with the out_file plugin to use this with an external service. I use this with fail2ban to do a simple IP based blocking for failed connection/login attempts (probably at some point we'll use a full SIEM system but this is what I currently got).

Does this mean that the external application outside the container collects logs from the symlink inside the container?

The external application monitors for failed logging attemps. So I actually need a single file which contains just the latest events. However, the external application is not able to reload the log files, i.e., I cannot use the buffer file directly since the hash is changing. Hence, I use the symlink to the buffer to readout the latest events.

If the symlink is a relative path, how does an external application read it?

It basically looks like this on the outside:

my_log -> ./buffer/my_bufferd_log_hash.log
my_log.20250415

If I keep using the absoulte path from fluentd this the link looks like this

my_log -> /fluentd/logs/my_buffered_log_hash.log
my_log.20250415

and this internal folder /fluentd/logs does not exist on the outside machine, hence I need to use the relative link above which works both on the inside and outside of the container.

The only thing I want is the my_log file. I don't even need the historical files. So using this plugin (and I know I this is not the original intent of the plugin) was the quickest solution for me and I thought using this relative link might be useful for others (i.e., in any case users wants to use this outside of the docker container or in a separate docker volume). In the long run I might even write a custom plugin (since I also want to correlate the logs for the both services) or will use an external SIEM system alltogether (but I think this is of no concern for the relative link here 😃 )

[daipom]

Thanks! I see!

Here's what I understand.

You mount a host directory to a container, and Fluentd inside that container writes files into that directory.
Then, Fail2ban on the host reads the symlink.
However, if the symlink is absolute, then you can not use it because it does not show the path on the host.
If the symlink is relative, then you can solve it on the host. (Do you expand this relative path in some way on the host side?)

I thought using this relative link might be useful for others (i.e., in any case users wants to use this outside of the docker container or in a separate docker volume)

Thanks so much!
I didn't assume this kind of use case for out_file, so I'm very grateful for it!

I have one question.
Can't we put the following symlink on the host side to resolve the absolute symlink?

/fluentd/logs -> (my_log directory path on the host)

(my_log -> /fluentd/logs/my_buffered_log_hash.log)

Even if we use a relative path, I think we will eventually need to resolve the relative path on the host side.
(Fail2ban can not read the relative path directly, right?)
Is it easier to place the symlink so that the host can resolve the absolute path?

[daipom]

If the symlink is relative, then you can solve it on the host. (Do you expand this relative path in some way on the host side?)

Even if we use a relative path, I think we will eventually need to resolve the relative path on the host side. (Fail2ban can not read the relative path directly, right?)

Sorry, I misunderstood a relative symlink behavior.
A relative path of the symlink is resolved based on the path of the symlink file.
So, maybe Fail2ban can read the relative symlink directly.

[daipom]

Thanks! Finally I understand!
This direction looks good to me!

It might be better to keep the default behavior.
If you want to use relative paths, please use the option function.

You mean keeping the behaviour of symlink_path as an absolute link and adding an additional option, e.g., relative_symlink_path = true/false to switch to relative links? I can certainly implement it like this.

Maybe it would not break compatibility if it were a relative path?
If so, I wonder if it should be an optional.
Some tests are failing, so we will need to check the cause.

[domna]

Maybe it would not break compatibility if it were a relative path?

I'm not sure. From my point of view it should not break anything. However, there might be a tricky edge case where this could be relevant.

If so, I wonder if it should be an optional.

For me it doesn't matter. I can certainly add a selection flag here.

Some tests are failing, so we will need to check the cause.

Yes, I did not look into them yet as I first wanted to discuss whether the approach is the way forward. Currently, I'm away but I'll look into it as soon as I'm back (on tuesday) and (hopefully) fix it.

So I'll proceed as follows:

• I'll add an optional flag for the relative path (so we can easier decide on both)
• I'll fix the test and maybe add relevant ones for the new relative path

[domna]

I added a new parameter symlink_path_use_relative to switch between absolute and relative symlink. It defaults to false mimicking the old behaviour. I also added a test for the relative symlink and the tests pass locally for me now.

[daipom]

@domna Thanks! Sorry for my late response.
This direction looks good to me!
I will check this soon.

[daipom]

Sorry for being late!
LGTM! Thanks!