Skip to content

Enable Filebeat's Journald input in all docker image variants #7995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

belimawr
Copy link
Contributor

@belimawr belimawr commented Apr 25, 2025

What does this PR do?

Filebeat's Journald input requires journalctl, so the systemd package is added to all docker image variants.

Why is it important?

Currently it is only possible to use integrations that requires the journald input with the complete docker image variant, this PR allows it to be used with any docker image variant. The complete docker image is the largest with more than 3Gb, while the standard Elastic-Agent image is less than 2Gb.

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

## Disruptive User Impact

How to test this PR locally

  1. Package the Elastic-Agent:
    DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="docker" PLATFORMS=linux/amd64 mage -v package
    
  2. Check the version of journalctl:
    docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent-complete-wolfi:9.1.0-SNAPSHOT  --version
    docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent-slim-wolfi:9.1.0-SNAPSHOT  --version
    docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent-slim:9.1.0-SNAPSHOT  --version
    docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-agent-ubi:9.1.0-SNAPSHOT  --version
    docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-otel-collector-wolfi:9.1.0-SNAPSHOT  --version
    docker run --rm -it --entrypoint journalctl docker.elastic.co/elastic-agent/elastic-otel-collector:9.1.0-SNAPSHOT  --version
    

Related issues

Questions to ask yourself

  • How are we going to support this in production?
  • How are we going to measure its adoption?
  • How are we going to debug this?
  • What are the metrics I should take care of?
  • ...

Filebeat's Journald input requires `journalctl`, so the systemd
package is added to all docker image variants.
Copy link
Contributor

mergify bot commented Apr 25, 2025

This pull request does not have a backport label. Could you fix it @belimawr? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

@belimawr belimawr added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-active-all Automated backport with mergify to all the active branches labels Apr 25, 2025
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

History

cc @belimawr

@belimawr belimawr marked this pull request as ready for review April 25, 2025 21:28
@belimawr belimawr requested a review from a team as a code owner April 25, 2025 21:28
@belimawr belimawr requested review from ycombinator and pchila April 25, 2025 21:28
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Apr 27, 2025
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@rdner
Copy link
Member

rdner commented Apr 28, 2025

@belimawr please add the image size comparison to the description.

@@ -73,15 +73,15 @@ ENV BEAT_SETUID_AS={{ .user }}
{{- if contains .from "ubi-minimal" }}
RUN for iter in {1..10}; do \
microdnf update -y && \
microdnf install -y tar gzip findutils shadow-utils ca-certificates gawk libcap xz && \
microdnf install -y tar gzip findutils shadow-utils ca-certificates gawk libcap xz systemd && \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much bigger does this make the images?

Also, we don't want it in every container, it would be in elastic-agent, elastic-agent-complete but not elastic-agent-slim. This is really dependent on how much the size increase is as we do not want them getting bigger since they are already huge.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The downside of using the elastic-agent-complete image is that people who don't use synthetics but want to read node logs on K8S now have to deploy a bunch of browser dependencies and such.

Perhaps this would be best addressed with an elastic-agent-journald container. We already have a lot of containers though so I'm not sure this clarifies anything with respect to which one someone should use.

Copy link
Member

@cmacknz cmacknz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we keep this (size dependent), it definitely shouldn't be in the slim variant at least.

microdnf clean all && \
exit_code=0 && break || exit_code=$? && echo "microdnf error: retry $iter in 10s" && sleep 10; done; (exit $exit_code)
{{- end }}

{{- if contains .from "wolfi" }}
RUN for iter in {1..10}; do \
apk fix && \
apk add --no-cache ca-certificates curl gawk shadow bash && \
apk add --no-cache ca-certificates curl gawk shadow bash systemd && \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't include systemd in wolfi because it is going to significantly broaden it's exposure to CVEs, the point of Wolfi is to minimize the OS level CVEs we are exposed to. Same rationale as Filebeat: https://github.com/elastic/beats/pull/44056/files#r2064612269

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-active-all Automated backport with mergify to all the active branches Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants