Use imfile instead of blackbox for file forwarding

[ctlong]

As of RSYSLOG v8.25.0, wildcards are allowed in imfile inputs. I believe that this means we can now replace blackbox with imfile inputs for this release.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[ctlong]

It was pointed out to me that recursive wildcards are what we actually need to replace blackbox, which RSYSLOG does not offer yet.

[ctlong]

Actually.. maybe it is allowed. Worth following up on.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.

[ctlong]

Re-opening this issue because I think specifying wildcard folder/file inputs is possible with rsyslog v8.25.0+.

e.g. I think we could specify /var/vcap/sys/log/*/*.log and that would be sufficient for imfile to find "[a]ll matching files in all matching subfolders" (link).

[ctlong]

Using imfile may not be as performant as blackbox + rsyslog, we'd probably want to check that out.

[rkoster]

Rsyslog is being installed in the stemcell here: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/da63b6fa6ec5f937ebfdbbebe4db923079d959eb/stemcell_builder/stages/base_ubuntu_packages/apply.sh#L56-L57

Also maybe relevant: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/issues?q=is%3Aopen+is%3Aissue+milestone%3A%2224.04+wishlist+%22

[bgandon]

Hi @ctlong, I was there at the Bosh meeting yesterday afternoon.

After checking on some old Bionic Stemcell and some recent plain Ubuntu Jammy box, I can confirm that the imfile.so module is included in the main rsyslog apt package.

Here on an old Bionic Stemcell, rsyslog is way recent enough:

$ dpkg -l rsyslog
...
||/ Name       Version                      Architecture  Description
+++-==========-============================-=============-=======================================
ii  rsyslog    8.2110.0-0adiscon1bionic1    amd64         a rocket-fast system for log processing

(and similarly on Jammy Stemcells)

And here is the imfile module:

$ dpkg -L rsyslog | grep imfile
/usr/lib/rsyslog/imfile.so

Rsyslog v8.25 was released on february 2017. As Bionic stemcells have always used the ppa:adiscon/v8-stable PPA with most recent v8 packages, it means that they all have the imfile module already installed by default.

About your suggestion on the *.log wildcard:

Re-opening this issue because I think specifying wildcard folder/file inputs is possible with rsyslog v8.25.0+.

e.g. I think we could specify /var/vcap/sys/log/*/*.log and that would be sufficient for imfile to find "[a]ll matching files in all matching subfolders" (link).

The answer is “no”, it would not be sufficient 🙂

Let me point you the already enforced logrotate config, that is a source of inspiration:

$ head -n4 /etc/logrotate.d/vcap 
# Generated by bosh-agent

/var/vcap/data/sys/log/*.log /var/vcap/data/sys/log/.*.log /var/vcap/data/sys/log/*/*.log /var/vcap/data/sys/log/*/.*.log /var/vcap/data/sys/log/*/*/*.log /var/vcap/data/sys/log/*/*/.*.log {
  missingok

I fact, this Logrotate config looks incorrect because the /var/vcap/sys symlink should be favored over the actual location /var/vcap/data/sys. This is a good opportunity to question and possibly fix this. /CC @ramonskie

In the end, following your suggestion, and staying consistent with the existing Logrotate config, Rsyslog would specify these wildcards:

/var/vcap/sys/log/*.log
/var/vcap/sys/log/.*.log
/var/vcap/sys/log/*/*.log
/var/vcap/sys/log/*/.*.log
/var/vcap/sys/log/*/*/*.log
/var/vcap/sys/log/*/*/.*.log

Hope this might help!

[ctlong]

That's great to hear! Thank you for tracking this down. My main takeaway is that, in potentially implementing this breaking change, we don't need to worry about:

• Updating the stemcell generation file, as it already includes the imfile module.
• Stemcell compatibility, as it has included the imfile module as far back as supported stemcells go.

In terms of the paths that we'd want to watch, I think some of the wildcard paths you laid out may be duplicative. Probably something to experiment with.

I also looked at the Configuration Locations in BOSH and found a few more paths that we might want to forward logs from:

• /var/vcap/bosh/log/current: Current Agent log. Agent's logs are logrotated and archives are kept in /var/vcap/bosh/log/ directory.
• /var/vcap/monit/monit.log: Monit activity log. Includes information about starts, stops, restarts, etc. of release job processes monitored by Monit.

Those two paths are not watched by blackbox, which means we might be ignoring them right now.

[bgandon]

In terms of the paths that we'd want to watch, I think some of the wildcard paths you laid out may be duplicative.

Well, these Logrotate wildcards are the source of truth for the base Bosh contract about logs within the /var/vcap/sys/log directory.

They are defined by the Bosh Agent here and to my experience, have evolved over time.

So, the contract is:

1. Filename must end with .log
2. Filename may start with a dot .
3. Allowed depth for such log files: 0 to 2 folders deep

(I’ve filed a PR to make it clear in the Bosh documentation, see cloudfoundry/docs-bosh#812)

That’s right, Monit and Agent logs are typically forgotten when it comes to forwarding all VMs logs! Certainly a good thing to become more exhaustive. Agent logs may contain sensitive data though, precisely because they are usually kept private to the Bosh instance. This should double-check this with the Bosh team.

[KarstenSchnitter]

Does using imfile plugin also read log files from the start? Currently blackbox is only tailing the log files. This means, that all log statements written when blackbox is not running are not forwarded. Hence, start and stop script logs are not forwarded. Adding the monit log file would also lead to very incomplete forwards when the file is only tailed.

Since #138 was closed in favour of this issue, I would want to know, whether imfile also supports a file marker as Fluent Bit does. So that it can continue reading log files, where it stop in the previous run. At least, when the file was not rotated in the mean time.

[chombium]

Hi @KarstenSchnitter, according to the docs, there shouldn't be log loss caused by restarts:

When rsyslogd is stopped while monitoring a text file, it records the last processed location and continues to work from there upon restart. So no data is lost during a restart (except, as noted above, if the file is rotated just in this very moment)