feat(agent): Add support for input probing

[LandonTClipp]

Summary

This PR adds support in RunningInput and the Agent to call an input plugin's Probe method if it exists and if the plugin has been configured for probing.

Checklist

• No AI generated code was used in this PR

Related issues

• Spec: docs(specs): Add probe as value to startup_error_behavior #16052
• inputs.nvidia-smi: Add config option to test a single run of nvidia-smi on plugin startup #15915
• probe_on_startup: Add input config and interface similar to startup_error_behavior #16028
• feat(inputs.nvidia_smi): Implement probing #16305

[srebhan]

Thanks @LandonTClipp for the PR! Some minor comments in the code, the most important one is to call Stop if probing fails to not leak connections or the like.

[LandonTClipp]

Blocked on #16364.

[Hipska]

I'm not sure this resolves #15915 as it doesn't add a Probe method for nvidia plugin?

[LandonTClipp]

I'm not sure this resolves #15915 as it doesn't add a Probe method for nvidia plugin?

You're right, I changed the verbiage.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

👍 This pull request doesn't change the Telegraf binary size

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[LandonTClipp]

@srebhan should be ready for another review.

[srebhan]

Thanks @LandonTClipp for your contribution!

[Hipska]

Could you give some comments on #16052 (review) please? As the spec is not really clear. Also a link to the spec in the PR message would be nice. (https://github.com/influxdata/telegraf/blob/master/docs/specs/tsd-009-probe-on-startup.md)

[Hipska]

#16052 (comment)

So starting from now (and when plugins implement it) Telegraf will need a restart in order for plugins to recover?

[LandonTClipp]

So starting from now (and when plugins implement it) Telegraf will need a restart in order for plugins to recover?

Starting from now, when plugins support it, and startup_error_behavior is set to probe, plugins will indeed be disabled for the lifetime of the process if they fail probing on startup.

[Hipska]

Why would you ever want that and have a plugin only recover when telegraf is restarted? I think I'm missing the use case here..

[LandonTClipp]

This particular use-case deals with environments running in a public cloud that I am helping to build. We distribute a custom build of Telegraf, along with associated config, that our users apt install from a public repo we maintain. It is hard to know what resources will be available in the customer environment ahead of time. One possibility is for the package to run a pre-install.sh script that introspects the resources and dynamically creates an appropriate Telegraf config file. Another is for the pre-install.sh to reach out to some metadata service to ask what should be installed and then dynamically create the config file. The other option, which this PR implements, says that Telegraf should just disable the plugin entirely if its associated resource is not available.

Note that this does not affect Telegraf's behavior if the resource existed at process startup but fails at some point after. In that case, Telegraf will continue to retry the plugin endlessly. The PR allows you to describe the case in which you want to assume that a resource will never exist if it doesn't exist at process startup.