Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zarf files archive path support #1962

Merged
merged 54 commits into from
Aug 27, 2023
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
d6e6976
WIP
cmwylie19 Aug 8, 2023
0ec1054
base work done
cmwylie19 Aug 9, 2023
5789d36
archive-path
cmwylie19 Aug 9, 2023
a46cc78
draft PR for review
cmwylie19 Aug 9, 2023
7cac2e5
fix(deps): update all non-major dependencies (#1866)
renovate[bot] Aug 9, 2023
9ea87d7
removed commented code
cmwylie19 Aug 9, 2023
b1e182e
Merge branch 'main' into 1928
cmwylie19 Aug 9, 2023
edcf11e
clean up code - prepare tests
cmwylie19 Aug 10, 2023
deb842b
implemented test
cmwylie19 Aug 10, 2023
56e069a
Merge branch 'main' into 1928
cmwylie19 Aug 10, 2023
67fc9fc
Error handling
cmwylie19 Aug 10, 2023
1baafd9
removed unneeded files and lint
cmwylie19 Aug 10, 2023
b7fe012
make docs-and-schema
cmwylie19 Aug 10, 2023
4e5299e
remove comments
cmwylie19 Aug 10, 2023
25d0f25
consistent error messaging
cmwylie19 Aug 10, 2023
7adacdc
Merge branch 'main' into 1928
cmwylie19 Aug 10, 2023
97000e6
update test for windows binary
cmwylie19 Aug 10, 2023
037d981
move order of tests
cmwylie19 Aug 10, 2023
5ece671
Update src/types/component.go
cmwylie19 Aug 10, 2023
04d51ff
Update src/pkg/utils/helpers/url.go
cmwylie19 Aug 10, 2023
4b5767c
Merge branch 'main' into 1928
cmwylie19 Aug 11, 2023
367f9c3
update e2e test
cmwylie19 Aug 10, 2023
42d168e
renamed function
cmwylie19 Aug 10, 2023
84f5add
refactor more efficiently
cmwylie19 Aug 11, 2023
8c5d0ae
refactor more efficiently
cmwylie19 Aug 11, 2023
1932629
updates for folders and files in folders
cmwylie19 Aug 12, 2023
23cc087
update e2e tests
cmwylie19 Aug 12, 2023
de9665a
remove archive-path example
cmwylie19 Aug 12, 2023
84f295c
resolve lang conflict
cmwylie19 Aug 12, 2023
ae3fe59
Merge branch 'main' into 1928
cmwylie19 Aug 12, 2023
d98e257
remove eks.yaml
cmwylie19 Aug 12, 2023
deb4ee8
update e2e test
cmwylie19 Aug 12, 2023
2473f5e
rm eks.yaml
cmwylie19 Aug 12, 2023
75dfa5e
logic for files
cmwylie19 Aug 12, 2023
bdecac3
rerun docs-and-schema
cmwylie19 Aug 12, 2023
ad951de
remove zarf-cache from test-archive e2e test
cmwylie19 Aug 12, 2023
ac14d5b
remove extra underscore in distros
cmwylie19 Aug 12, 2023
f277275
update path for zarf-package-distro-eks-multi-0.0.2.tar.zst
cmwylie19 Aug 14, 2023
3e4c4b0
lint and recursive delete
cmwylie19 Aug 14, 2023
6d51ee9
Merge branch 'main' into 1928
cmwylie19 Aug 15, 2023
6a330e6
remove duplicated code
cmwylie19 Aug 15, 2023
284c703
disable bb
cmwylie19 Aug 15, 2023
98128cf
address updates from team
cmwylie19 Aug 21, 2023
29b3c94
Merge branch 'main' into 1928
cmwylie19 Aug 21, 2023
d66232d
make docs-and-schema
cmwylie19 Aug 21, 2023
6bba743
update test
cmwylie19 Aug 22, 2023
8866d44
clean up duplicated code, address comments
cmwylie19 Aug 22, 2023
9f9b71e
Update src/test/e2e/00_use_cli_test.go
cmwylie19 Aug 24, 2023
e048b71
Update src/pkg/packager/create.go
cmwylie19 Aug 24, 2023
1f7ae70
Update src/pkg/packager/create.go
cmwylie19 Aug 24, 2023
17e965a
Update src/pkg/packager/create.go
cmwylie19 Aug 24, 2023
1545a53
Merge branch 'main' into 1928
cmwylie19 Aug 25, 2023
d657a79
Merge branch 'main' into 1928
Racer159 Aug 25, 2023
cf561b5
Merge branch 'main' into 1928
Racer159 Aug 26, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions examples/archive-path/zarf.yaml
cmwylie19 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
kind: ZarfPackageConfig
metadata:
name: archive-path
description: "Test archive path"


components:
- name: download-flux
required: true
files:
- source: https://github.com/fluxcd/flux2/releases/download/v2.0.1/flux_2.0.1_darwin_arm64.tar.gz
# target: /Users/cmwylie19/zarf/flux
target: flux
executable: true
shasum: 520f25324d42e222ccdc75009659452808adcbecb2b744f2ebf037e14e18cd69
archivePath: flux
- name: flux-version
required: true
actions:
onDeploy:
before:
- cmd: ./flux -v
description: What's the version
mute: false
- cmd: rm flux
description: Remove the zipped

1 change: 1 addition & 0 deletions src/config/lang/english.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ const (
ErrCreatingDir = "failed to create directory %s: %s"
ErrRemoveFile = "failed to remove file %s: %s"
ErrUnarchive = "failed to unarchive %s: %s"
ErrFilename = "failed to extract filename from url %s: %s"
)

// Zarf CLI commands.
Expand Down
2 changes: 1 addition & 1 deletion src/pkg/packager/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import (
"strings"

"github.com/AlecAivazis/survey/v2"
"github.com/Masterminds/semver/v3"
"github.com/Masterminds/semver"
"github.com/defenseunicorns/zarf/src/config/lang"
"github.com/defenseunicorns/zarf/src/internal/cluster"
"github.com/defenseunicorns/zarf/src/internal/packager/sbom"
Expand Down
37 changes: 37 additions & 0 deletions src/pkg/packager/create.go
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,34 @@ func (p *Packager) addComponent(index int, component types.ZarfComponent, isSkel
if err := utils.DownloadToFile(file.Source, dst, component.CosignKeyPath); err != nil {
return fmt.Errorf(lang.ErrDownloading, file.Source, err.Error())
}
// Verify we can decompress the file
if file.ArchivePath != "" || helpers.SupportedCompressionFormat(file.Source) {
// extract filename from target
dstFileName, err := helpers.ExtractFilenameFromURL(file.Source)
if err != nil {
message.Fatalf(err, lang.ErrFilename, file.Source, err.Error())
}
// For archiver support, we need to rename the file to the original
newDst, err := helpers.RenamePathWithFilename(dst, dstFileName)
if err != nil {
fmt.Printf("Could not renamePathWithFilename %s, %s, %s", file.Target, dst, err.Error())
}

err = os.Rename(dst, newDst)
if err != nil {
return fmt.Errorf(lang.ErrWritingFile, dst, err)
}

// Decompress the file into the target directory
err = archiver.Unarchive(newDst, helpers.GetDirFromFilename(dst))
if err != nil {
message.Fatalf(err, lang.CmdToolsArchiverDecompressErr, err.Error())
}

// for Sha256sum rename dst
dst = newDst

}
} else {
if err := utils.CreatePathAndCopy(file.Source, dst); err != nil {
return fmt.Errorf("unable to copy file %s: %w", file.Source, err)
Expand All @@ -397,6 +425,15 @@ func (p *Packager) addComponent(index int, component types.ZarfComponent, isSkel
if actualShasum, _ := utils.GetCryptoHashFromFile(dst, crypto.SHA256); actualShasum != file.Shasum {
return fmt.Errorf("shasum mismatch for file %s: expected %s, got %s", file.Source, file.Shasum, actualShasum)
cmwylie19 marked this conversation as resolved.
Show resolved Hide resolved
}

if file.ArchivePath != "" {
// Remove all files in directory other file
err = helpers.KeepOnlyFiles(filepath.Dir(dst), []string{file.ArchivePath, filepath.Base(dst)})
if err != nil {
fmt.Println("Error cleaning up directory", err.Error())
}
}

}

if file.Executable || utils.IsDir(dst) {
Expand Down
8 changes: 8 additions & 0 deletions src/pkg/packager/deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -319,10 +319,18 @@ func (p *Packager) processComponentFiles(component types.ZarfComponent, pkgLocat

// If a shasum is specified check it again on deployment as well
if file.Shasum != "" {
tempFileLocation := fileLocation
if file.ArchivePath != "" {
fileSource := filepath.Base(file.Source)

fileLocation, _ = helpers.RenamePathWithFilename(fileLocation, fileSource)
}

spinner.Updatef("Validating SHASUM for %s", file.Target)
if shasum, _ := utils.GetCryptoHashFromFile(fileLocation, crypto.SHA256); shasum != file.Shasum {
return fmt.Errorf("shasum mismatch for file %s: expected %s, got %s", file.Source, file.Shasum, shasum)
}
fileLocation = tempFileLocation
}

// Replace temp target directory and home directory
Expand Down
20 changes: 20 additions & 0 deletions src/pkg/utils/helpers/compression.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: 2021-Present The Zarf Authors

// Package helpers provides generic helper functions with no external imports
package helpers

import (
"strings"
)

func SupportedCompressionFormat(filename string) bool {

Check warning on line 11 in src/pkg/utils/helpers/compression.go

View workflow job for this annotation

GitHub Actions / validate

exported function SupportedCompressionFormat should have comment or be unexported
supportedFormats := []string{".tar.gz", ".br", ".bz2", ".zip", ".lz4", ".sz", ".xz", ".zz", ".zst"}

for _, format := range supportedFormats {
if strings.HasSuffix(filename, format) {
return true
}
}
return false
}
53 changes: 53 additions & 0 deletions src/pkg/utils/helpers/files.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
package helpers

import (
"fmt"
"net/url"
"os"
"path"
"path/filepath"
)

func KeepOnlyFiles(directory string, fileNames []string) error {

Check warning on line 11 in src/pkg/utils/helpers/files.go

View workflow job for this annotation

GitHub Actions / validate

exported function KeepOnlyFiles should have comment or be unexported
files, err := os.ReadDir(directory)
if err != nil {
return err
}

filesToKeep := make(map[string]bool)
for _, fileName := range fileNames {
filesToKeep[fileName] = true
}

for _, file := range files {
filePath := filepath.Join(directory, file.Name())
if !filesToKeep[file.Name()] {
err := os.RemoveAll(filePath)
if err != nil {
return err
}
fmt.Println("Deleted:", filePath)
}
}

return nil
}

func GetDirFromFilename(target string) string {

Check warning on line 36 in src/pkg/utils/helpers/files.go

View workflow job for this annotation

GitHub Actions / validate

exported function GetDirFromFilename should have comment or be unexported
return filepath.Dir(target)
}
func RenamePathWithFilename(target, fileName string) (string, error) {

Check warning on line 39 in src/pkg/utils/helpers/files.go

View workflow job for this annotation

GitHub Actions / validate

exported function RenamePathWithFilename should have comment or be unexported
dir := filepath.Dir(target)
newPath := filepath.Join(dir, fileName)
return newPath, nil
}

func ExtractFilenameFromURL(urlStr string) (string, error) {

Check warning on line 45 in src/pkg/utils/helpers/files.go

View workflow job for this annotation

GitHub Actions / validate

exported function ExtractFilenameFromURL should have comment or be unexported
parsedURL, err := url.Parse(urlStr)
if err != nil {
return "", err
}

filename := path.Base(parsedURL.Path)
return filename, nil
}
11 changes: 6 additions & 5 deletions src/types/component.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,12 @@ type ZarfComponentOnlyCluster struct {

// ZarfFile defines a file to deploy.
type ZarfFile struct {
Source string `json:"source" jsonschema:"description=Local folder or file path or remote URL to pull into the package"`
Shasum string `json:"shasum,omitempty" jsonschema:"description=(files only) Optional SHA256 checksum of the file"`
Target string `json:"target" jsonschema:"description=The absolute or relative path where the file or folder should be copied to during package deploy"`
Executable bool `json:"executable,omitempty" jsonschema:"description=(files only) Determines if the file should be made executable during package deploy"`
Symlinks []string `json:"symlinks,omitempty" jsonschema:"description=List of symlinks to create during package deploy"`
Source string `json:"source" jsonschema:"description=Local folder or file path or remote URL to pull into the package"`
Shasum string `json:"shasum,omitempty" jsonschema:"description=(files only) Optional SHA256 checksum of the file"`
Target string `json:"target" jsonschema:"description=The absolute or relative path where the file or folder should be copied to during package deploy"`
Executable bool `json:"executable,omitempty" jsonschema:"description=(files only) Determines if the file should be made executable during package deploy"`
Symlinks []string `json:"symlinks,omitempty" jsonschema:"description=List of symlinks to create during package deploy"`
ArchivePath string `json:"archivePath,omitempty" jsonschema:"description=Local folder or file to be extracted into the package"`
cmwylie19 marked this conversation as resolved.
Show resolved Hide resolved
}

// ZarfChart defines a helm chart to be deployed.
Expand Down
Loading