Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add error validation for update all #618

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
3750a02
propogate error
ashnamehrotra Apr 5, 2024
d5a1059
do not need to set unassigned updates to nil
ashnamehrotra Apr 5, 2024
5f7e590
specify linux for OS in ExtractFileFromState and get archi
ashnamehrotra Apr 5, 2024
a8a6f57
save debugging
ashnamehrotra Apr 5, 2024
b4aa1e3
take OSVersion as pkgmgr arg
ashnamehrotra Apr 9, 2024
3c1c92e
working debian changes
ashnamehrotra Apr 9, 2024
a65c173
update vex unit tests for pkgmgr changes
ashnamehrotra Apr 9, 2024
5a037d3
saving part of debian changes
ashnamehrotra Apr 19, 2024
eefb00f
debian changes to read only outdated packages via packages.txt
ashnamehrotra Apr 23, 2024
c8031c8
cleanup
ashnamehrotra Apr 23, 2024
c0a9904
rpm changes - read from manifest2 and compare w yum info
ashnamehrotra Apr 23, 2024
843f264
cleanup
ashnamehrotra Apr 23, 2024
435fab5
handle platform OS case, more cleanup
ashnamehrotra Apr 23, 2024
aae6520
modify integration test to include distroless
ashnamehrotra Apr 23, 2024
51e2066
Merge branch 'main' of github.com:project-copacetic/copacetic into pa…
ashnamehrotra Apr 23, 2024
84e90f3
clarifications
ashnamehrotra Apr 23, 2024
4327c96
test all but custom dpkg/status.d
ashnamehrotra Apr 24, 2024
cfe7f56
rename test file
ashnamehrotra Apr 24, 2024
c6face3
run go mod tidy
ashnamehrotra Apr 24, 2024
e6cb18b
revert ignore errors test case
ashnamehrotra Apr 24, 2024
8d33292
fix description - fluent bit container does have apt
ashnamehrotra Apr 24, 2024
c89f379
fix unit test dpkg
ashnamehrotra Apr 24, 2024
3d96ff2
lint fix
ashnamehrotra Apr 24, 2024
c42224b
update golang.org/x/net to v0.23.0
ashnamehrotra Apr 24, 2024
6855c3a
downloadPath to dpkgDownloadPath const
ashnamehrotra Apr 24, 2024
ab28700
check for .md5sums file
ashnamehrotra Apr 24, 2024
611d610
patch test changes to test same set of images with ignore-errors false
ashnamehrotra Apr 24, 2024
64763ff
add kube-proxy v1.26.1 to integration test
ashnamehrotra Apr 24, 2024
8149583
libss1 to libssl1
ashnamehrotra Apr 25, 2024
e1284e3
add documentation
ashnamehrotra Apr 25, 2024
a0099f4
Merge branch 'main' of github.com:project-copacetic/copacetic into pa…
ashnamehrotra Apr 25, 2024
3cd0953
GetPackageInfo unit test
ashnamehrotra Apr 26, 2024
6038534
space formatting txt block
ashnamehrotra Apr 26, 2024
88656d4
getOsVersion unit test
ashnamehrotra Apr 26, 2024
04bd6bf
add darwin comment
ashnamehrotra Apr 26, 2024
c1081b0
update quickstart w documentation
ashnamehrotra Apr 27, 2024
8eaf7b6
Merge branch 'main' of github.com:project-copacetic/copacetic into pa…
ashnamehrotra Apr 27, 2024
2dbc3de
dpkg validation update all regular and distroless
ashnamehrotra May 10, 2024
ef48379
add error_log.txt
ashnamehrotra May 10, 2024
8fa43a1
rpm validation update all regular and distroless
ashnamehrotra May 10, 2024
98f89fd
apk validation update all regular
ashnamehrotra May 10, 2024
2b79b9a
change bash to sh dpkg
ashnamehrotra May 10, 2024
3666ec3
check for error to filter out warning
ashnamehrotra May 10, 2024
b860875
revert apk changes
ashnamehrotra May 10, 2024
fb28612
use llb args instead
ashnamehrotra May 10, 2024
7c4f2df
apk changes
ashnamehrotra May 10, 2024
4b99e52
Merge branch 'main' of github.com:project-copacetic/copacetic into pa…
ashnamehrotra May 10, 2024
800db97
output fixes
ashnamehrotra May 10, 2024
d5fe32b
fix microdnf
ashnamehrotra May 10, 2024
f8d65f4
try sh instead of bash
ashnamehrotra May 10, 2024
1245ae8
Merge branch 'main' of github.com:project-copacetic/copacetic into pa…
ashnamehrotra May 10, 2024
92718bb
fix if statement
ashnamehrotra May 21, 2024
cfa7e6e
Merge branch 'main' of github.com:project-copacetic/copacetic into pa…
ashnamehrotra Jun 12, 2024
c57f9e0
create and use util function for sh c
ashnamehrotra Jun 25, 2024
22ef97d
use util func
ashnamehrotra Jun 25, 2024
ddf1e4e
Merge branch 'main' of github.com:project-copacetic/copacetic into pa…
ashnamehrotra Jun 25, 2024
20a3290
Merge branch 'main' into patch-no-scanner-distroless-2
sozercan Jun 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions pkg/buildkit/buildkit.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,10 @@
})
}

func Sh(cmd string) llb.RunOption {
return llb.Args([]string{"/bin/sh", "-c", cmd})

Check warning on line 94 in pkg/buildkit/buildkit.go

View check run for this annotation

Codecov / codecov/patch

pkg/buildkit/buildkit.go#L93-L94

Added lines #L93 - L94 were not covered by tests
}

func ArrayFile(input []string) []byte {
var b bytes.Buffer
for _, s := range input {
Expand Down
15 changes: 10 additions & 5 deletions pkg/pkgmgr/apk.go
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@
func (am *apkManager) InstallUpdates(ctx context.Context, manifest *unversioned.UpdateManifest, ignoreErrors bool) (*llb.State, []string, error) {
// If manifest is nil, update all packages
if manifest == nil {
updatedImageState, _, err := am.upgradePackages(ctx, nil)
updatedImageState, _, err := am.upgradePackages(ctx, nil, ignoreErrors)

Check warning on line 123 in pkg/pkgmgr/apk.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/apk.go#L123

Added line #L123 was not covered by tests
if err != nil {
return updatedImageState, nil, err
}
Expand All @@ -140,7 +140,7 @@
}
log.Debugf("latest unique APKs: %v", updates)

updatedImageState, resultsBytes, err := am.upgradePackages(ctx, updates)
updatedImageState, resultsBytes, err := am.upgradePackages(ctx, updates, ignoreErrors)

Check warning on line 143 in pkg/pkgmgr/apk.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/apk.go#L143

Added line #L143 was not covered by tests
if err != nil {
return nil, nil, err
}
Expand All @@ -161,7 +161,7 @@
// TODO: support "distroless" Alpine images (e.g. APKO images)
// Still assumes that APK exists in the target image and is pathed, which can be addressed by
// mounting a copy of apk-tools-static into the image and invoking apk-static directly.
func (am *apkManager) upgradePackages(ctx context.Context, updates unversioned.UpdatePackages) (*llb.State, []byte, error) {
func (am *apkManager) upgradePackages(ctx context.Context, updates unversioned.UpdatePackages, ignoreErrors bool) (*llb.State, []byte, error) {

Check warning on line 164 in pkg/pkgmgr/apk.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/apk.go#L164

Added line #L164 was not covered by tests
// TODO: Add support for custom APK config
apkUpdated := am.config.ImageState.Run(llb.Shlex("apk update"), llb.WithProxy(utils.GetProxy()), llb.IgnoreCache).Root()

Expand Down Expand Up @@ -200,8 +200,13 @@
}
} else {
// if updates is not specified, update all packages
installCmd := `apk upgrade --no-cache`
apkInstalled = apkUpdated.Run(llb.Shlex(installCmd), llb.WithProxy(utils.GetProxy())).Root()
installCmd := `output=$(apk upgrade --no-cache 2>&1); if [ $? -ne 0 ]; then echo "$output" >>error_log.txt; fi`
apkInstalled = apkUpdated.Run(buildkit.Sh(installCmd), llb.WithProxy(utils.GetProxy())).Root()

Check warning on line 204 in pkg/pkgmgr/apk.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/apk.go#L203-L204

Added lines #L203 - L204 were not covered by tests

// Validate no errors were encountered if updating all
if !ignoreErrors {
apkInstalled = apkInstalled.Run(buildkit.Sh("if [ -s error_log.txt ]; then cat error_log.txt; exit 1; fi")).Root()

Check warning on line 208 in pkg/pkgmgr/apk.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/apk.go#L207-L208

Added lines #L207 - L208 were not covered by tests
}
}

// Diff the installed updates and merge that into the target image
Expand Down
39 changes: 28 additions & 11 deletions pkg/pkgmgr/dpkg.go
Original file line number Diff line number Diff line change
Expand Up @@ -122,19 +122,17 @@
// If manifest nil, update all packages
if manifest == nil {
if dm.isDistroless {
updatedImageState, _, err := dm.unpackAndMergeUpdates(ctx, nil, toolImageName)
updatedImageState, _, err := dm.unpackAndMergeUpdates(ctx, nil, toolImageName, ignoreErrors)

Check warning on line 125 in pkg/pkgmgr/dpkg.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/dpkg.go#L125

Added line #L125 was not covered by tests
if err != nil {
return updatedImageState, nil, err
}
// add validation in the future
return updatedImageState, nil, nil
}

updatedImageState, _, err := dm.installUpdates(ctx, nil)
updatedImageState, _, err := dm.installUpdates(ctx, nil, ignoreErrors)

Check warning on line 132 in pkg/pkgmgr/dpkg.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/dpkg.go#L132

Added line #L132 was not covered by tests
if err != nil {
return updatedImageState, nil, err
}
// add validation in the future
return updatedImageState, nil, nil
}

Expand All @@ -153,12 +151,12 @@
var updatedImageState *llb.State
var resultManifestBytes []byte
if dm.isDistroless {
updatedImageState, resultManifestBytes, err = dm.unpackAndMergeUpdates(ctx, updates, toolImageName)
updatedImageState, resultManifestBytes, err = dm.unpackAndMergeUpdates(ctx, updates, toolImageName, ignoreErrors)

Check warning on line 154 in pkg/pkgmgr/dpkg.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/dpkg.go#L154

Added line #L154 was not covered by tests
if err != nil {
return nil, nil, err
}
} else {
updatedImageState, resultManifestBytes, err = dm.installUpdates(ctx, updates)
updatedImageState, resultManifestBytes, err = dm.installUpdates(ctx, updates, ignoreErrors)

Check warning on line 159 in pkg/pkgmgr/dpkg.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/dpkg.go#L159

Added line #L159 was not covered by tests
if err != nil {
return nil, nil, err
}
Expand Down Expand Up @@ -304,7 +302,7 @@
//
// TODO: Support Debian images with valid dpkg status but missing tools. No current examples exist in test set
// i.e. extra RunOption to mount a copy of busybox-static or full apt install into the image and invoking that.
func (dm *dpkgManager) installUpdates(ctx context.Context, updates unversioned.UpdatePackages) (*llb.State, []byte, error) {
func (dm *dpkgManager) installUpdates(ctx context.Context, updates unversioned.UpdatePackages, ignoreErrors bool) (*llb.State, []byte, error) {

Check warning on line 305 in pkg/pkgmgr/dpkg.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/dpkg.go#L305

Added line #L305 was not covered by tests
// TODO: Add support for custom APT config and gpg key injection
// Since this takes place in the target container, it can interfere with install actions
// such as the installation of the updated debian-archive-keyring package, so it's probably best
Expand All @@ -330,11 +328,16 @@
installCmd = fmt.Sprintf(aptInstallTemplate, strings.Join(pkgStrings, " "))
} else {
// if updates is not specified, update all packages
installCmd = `sh -c "apt upgrade -y && apt clean -y && apt autoremove"`
installCmd = `sh -c "output=$(apt upgrade -y && apt clean -y && apt autoremove 2>&1); if [ $? -ne 0 ]; then echo "$output" >>error_log.txt; fi"`

Check warning on line 331 in pkg/pkgmgr/dpkg.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/dpkg.go#L331

Added line #L331 was not covered by tests
}

aptInstalled := aptUpdated.Run(llb.Shlex(installCmd), llb.WithProxy(utils.GetProxy())).Root()

// Validate no errors were encountered if updating all
if updates == nil && !ignoreErrors {
aptInstalled = aptInstalled.Run(buildkit.Sh("if [ -s error_log.txt ]; then cat error_log.txt; exit 1; fi")).Root()

Check warning on line 338 in pkg/pkgmgr/dpkg.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/dpkg.go#L337-L338

Added lines #L337 - L338 were not covered by tests
}

// Write results.manifest to host for post-patch validation
const outputResultsTemplate = `sh -c 'grep "^Package:\|^Version:" "%s" >> "%s"'`
outputResultsCmd := fmt.Sprintf(outputResultsTemplate, dpkgStatusPath, resultManifest)
Expand All @@ -352,7 +355,7 @@
return &patchMerge, resultsBytes, nil
}

func (dm *dpkgManager) unpackAndMergeUpdates(ctx context.Context, updates unversioned.UpdatePackages, toolImage string) (*llb.State, []byte, error) {
func (dm *dpkgManager) unpackAndMergeUpdates(ctx context.Context, updates unversioned.UpdatePackages, toolImage string, ignoreErrors bool) (*llb.State, []byte, error) {

Check warning on line 358 in pkg/pkgmgr/dpkg.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/dpkg.go#L358

Added line #L358 was not covered by tests
imagePlatform, err := dm.config.ImageState.GetPlatform(ctx)
if err != nil {
return nil, nil, fmt.Errorf("unable to get image platform %w", err)
Expand Down Expand Up @@ -415,11 +418,25 @@
}
downloadCmd = fmt.Sprintf(aptDownloadTemplate, strings.Join(pkgStrings, " "))
} else {
// only updated the outdated pacakges from packages.txt
downloadCmd = "xargs -a packages.txt -n 1 apt download --no-install-recommends"
// only update the outdated pacakges from packages.txt
downloadCmd = `
packages=$(<packages.txt)
for package in $packages; do
output=$(apt download --no-install-recommends "$package" 2>&1)
if [ $? -ne 0 ]; then
echo "$output" >>error_log.txt
fi
done
`

Check warning on line 430 in pkg/pkgmgr/dpkg.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/dpkg.go#L422-L430

Added lines #L422 - L430 were not covered by tests
}

downloaded := updated.Dir(dpkgDownloadPath).Run(llb.Args([]string{"bash", "-c", downloadCmd}), llb.WithProxy(utils.GetProxy())).Root()

// Validate no errors were encountered if updating all
if updates == nil && !ignoreErrors {
downloaded = downloaded.Run(buildkit.Sh("if [ -s error_log.txt ]; then cat error_log.txt; exit 1; fi")).Root()

Check warning on line 437 in pkg/pkgmgr/dpkg.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/dpkg.go#L436-L437

Added lines #L436 - L437 were not covered by tests
}

diffState := llb.Diff(updated, downloaded)

// Scripted enumeration and dpkg unpack of all downloaded packages [layer to merge with target]
Expand Down
36 changes: 27 additions & 9 deletions pkg/pkgmgr/rpm.go
Original file line number Diff line number Diff line change
Expand Up @@ -211,12 +211,12 @@
var updatedImageState *llb.State
var resultManifestBytes []byte
if rm.isDistroless {
updatedImageState, resultManifestBytes, err = rm.unpackAndMergeUpdates(ctx, updates, toolImageName)
updatedImageState, resultManifestBytes, err = rm.unpackAndMergeUpdates(ctx, updates, toolImageName, ignoreErrors)

Check warning on line 214 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L214

Added line #L214 was not covered by tests
if err != nil {
return nil, nil, err
}
} else {
updatedImageState, resultManifestBytes, err = rm.installUpdates(ctx, updates)
updatedImageState, resultManifestBytes, err = rm.installUpdates(ctx, updates, ignoreErrors)

Check warning on line 219 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L219

Added line #L219 was not covered by tests
if err != nil {
return nil, nil, err
}
Expand Down Expand Up @@ -383,7 +383,7 @@
//
// TODO: Support RPM-based images with valid rpm status but missing tools. (e.g. calico images > v3.21.0)
// i.e. extra RunOption to mount a copy of rpm tools installed into the image and invoking that.
func (rm *rpmManager) installUpdates(ctx context.Context, updates unversioned.UpdatePackages) (*llb.State, []byte, error) {
func (rm *rpmManager) installUpdates(ctx context.Context, updates unversioned.UpdatePackages, ignoreErrors bool) (*llb.State, []byte, error) {

Check warning on line 386 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L386

Added line #L386 was not covered by tests
pkgs := ""

// If specific updates, provided, parse into pkg names, else will update all
Expand All @@ -400,20 +400,25 @@
var installCmd string
switch {
case rm.rpmTools["dnf"] != "":
const dnfInstallTemplate = `sh -c '%[1]s upgrade %[2]s -y && %[1]s clean all'`
const dnfInstallTemplate = `sh -c 'output=$(%[1]s upgrade %[2]s -y && %[1]s clean all 2>&1); if [ $? -ne 0 ]; then echo "$output" >>error_log.txt; fi'`

Check warning on line 403 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L403

Added line #L403 was not covered by tests
installCmd = fmt.Sprintf(dnfInstallTemplate, rm.rpmTools["dnf"], pkgs)
case rm.rpmTools["yum"] != "":
const yumInstallTemplate = `sh -c '%[1]s upgrade %[2]s -y && %[1]s clean all'`
const yumInstallTemplate = `sh -c 'output=$(%[1]s upgrade %[2]s -y && %[1]s clean all 2>&1); if [ $? -ne 0 ]; then echo "$output" >>error_log.txt; fi'`

Check warning on line 406 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L406

Added line #L406 was not covered by tests
installCmd = fmt.Sprintf(yumInstallTemplate, rm.rpmTools["yum"], pkgs)
case rm.rpmTools["microdnf"] != "":
const microdnfInstallTemplate = `sh -c '%[1]s update %[2]s && %[1]s clean all'`
const microdnfInstallTemplate = `sh -c 'output=$(%[1]s update %[2]s && %[1]s clean all 2>&1); if [ $? -ne 0 ]; then echo "$output" >>error_log.txt; fi'`

Check warning on line 409 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L409

Added line #L409 was not covered by tests
installCmd = fmt.Sprintf(microdnfInstallTemplate, rm.rpmTools["microdnf"], pkgs)
default:
err := errors.New("unexpected: no package manager tools were found for patching")
return nil, nil, err
}
installed := rm.config.ImageState.Run(llb.Shlex(installCmd), llb.WithProxy(utils.GetProxy())).Root()

// Validate no errors were encountered if updating all
if updates == nil && !ignoreErrors {
installed = installed.Run(buildkit.Sh("if [ -s error_log.txt ]; then cat error_log.txt; exit 1; fi")).Root()

Check warning on line 419 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L418-L419

Added lines #L418 - L419 were not covered by tests
}

// Write results.manifest to host for post-patch validation
var resultBytes []byte
var err error
Expand All @@ -434,7 +439,7 @@
return &patchMerge, resultBytes, nil
}

func (rm *rpmManager) unpackAndMergeUpdates(ctx context.Context, updates unversioned.UpdatePackages, toolImage string) (*llb.State, []byte, error) {
func (rm *rpmManager) unpackAndMergeUpdates(ctx context.Context, updates unversioned.UpdatePackages, toolImage string, ignoreErrors bool) (*llb.State, []byte, error) {

Check warning on line 442 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L442

Added line #L442 was not covered by tests
// Spin up a build tooling container to fetch and unpack packages to create patch layer.
// Pull family:version -> need to create version to base image map
toolingBase := llb.Image(toolImage,
Expand Down Expand Up @@ -489,10 +494,23 @@
downloadCmd = fmt.Sprintf(rpmDownloadTemplate, strings.Join(pkgStrings, " "))
} else {
// only updated the outdated pacakges from packages.txt
downloadCmd = `xargs -a packages.txt -n 1 yumdownloader --downloadonly --downloaddir=. --best -y`
downloadCmd = `
packages=$(<packages.txt)
for package in $packages; do
output=$(yumdownloader --downloadonly --downloaddir=. --best -y "$package" 2>&1)
if [ $? -ne 0 ]; then
echo "$output" >>error_log.txt
fi
done
`

Check warning on line 505 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L497-L505

Added lines #L497 - L505 were not covered by tests
}

downloaded := busyboxCopied.Run(llb.Shlex(downloadCmd), llb.WithProxy(utils.GetProxy())).Root()
downloaded := busyboxCopied.Run(buildkit.Sh(downloadCmd), llb.WithProxy(utils.GetProxy())).Root()

Check warning on line 508 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L508

Added line #L508 was not covered by tests

// Validate no errors were encountered if updating all
if updates == nil && !ignoreErrors {
downloaded = downloaded.Run(buildkit.Sh("if [ -s error_log.txt ]; then cat error_log.txt; exit 1; fi")).Root()

Check warning on line 512 in pkg/pkgmgr/rpm.go

View check run for this annotation

Codecov / codecov/patch

pkg/pkgmgr/rpm.go#L511-L512

Added lines #L511 - L512 were not covered by tests
}

// Scripted enumeration and rpm install of all downloaded packages under the download folder as root
const extractTemplate = `sh -c 'for f in %[1]s/*.rpm ; do rpm2cpio "$f" | cpio -idmv -D %[1]s ; done'`
Expand Down
Loading