Update (#1559)

Add amz Linux 2023 and update testing scripts

.gitignore

@@ -5,6 +5,7 @@
 *.ovf
 *.vmdk
 *.vdi
+*.qcow2
 *.log
 virtualfloppy.vfd
 packer_cache/

AMZ_README_FIRST.md

@@ -1,12 +1,12 @@
-This is not your normal Bento box. Instead of building a system from an ISO we're building a system from an Amazon provided VirtualBox VDI file. This means the process is a bit different than usual.
+This is not your normal Bento box. Instead of building a system from an ISO we're building a system from an Amazon provided vm hdd image files. This means the process is a bit different than usual.
 
 # Building this box
 
-Simply run the AMZ_build_virtualbox-ovf.sh script
+Simply run one of the amazonlinxu-2*-build.sh scripts
 
-This script will:
+These scripts will:
 
-1. Download the VirtualBox .vdi file for Amazon Linux 2 and place it in the amz_working_files directory. Amazon hosts these at <https://cdn.amazonlinux.com/os-images/latest/virtualbox/>. It will name it amazon.vdi instead of the version specific name that Amazon gives it on their site
-1. It will prepare this VDI file for packer and export it as a OVF file
+1. Download the vm image file for Amazon Linux 2 or 2023 and place it in the amz_working_files directory. Amazon hosts these at <https://cdn.amazonlinux.com/os-images/latest/> and <https://cdn.amazonlinux.com/al2023/os-images/latest/>.
+1. It will prepare the VDI file for packer and export it as a OVF file
 1. It will run the packer build
-1. Lastly it will clean up the leftover files in the working directory
+1. Lastly it will clean up the leftover files in the working directory on successful completion

CHANGELOG.md

@@ -52,9 +52,21 @@ Markdown table generated at <https://www.tablesgenerator.com/markdown_tables#>
 - Look into making all build uefi default builds
 - Create CD pipeline to upload vagrant boxes after PR is merged
 - Create CD pipeline to build and upload new versions of vagrant boxes once every 3 months with the latest patches
-- Added ability to configure bento upload of private or public boxes in builds.yml, defaults to private
 
-## [unreleased] (2024-02-16)
+## [unreleased] (2024-03-20)
+
+## [v4.0.2] (2024-03-20)
+
+- Update upload message to show architecture
+- Fix open-vm-tools install for vmware builds
+- Fix amazonlinux 2 build script
+- Added amazonlinux 2023 build scripts
+- Allow failed tests to continue and report at the end each test that failed
+- Added amazonlinux aarch64 packer templates
+- removed unneeded config from amazon user-data script for seed.iso
+- Added additional qemu variables for various qemu source properties
+- Added virtualbox-ovf vbox_source_path and vbox_checksum variables
+- Added recursive flag to rhel cleanup scripts for persistent network files
 
 ## [v4.0.1] (2024-02-16)
 
@@ -108,6 +120,7 @@ Markdown table generated at <https://www.tablesgenerator.com/markdown_tables#>
 - Updated Bento to continue builds even if one fails and report all failures at the end and exit with exit code 1
 - Added qemu clone of libvirt box when metadata file is created for uploading to vagrant qemu provider
 - Made build shell scripts customizable through packer variable
+- Added ability to configure bento upload of private or public boxes in builds.yml, defaults to private
 
 ## [v3.1.1] (2023-07-07)
 

README.md

@@ -4,7 +4,7 @@ Bento is a project that encapsulates [Packer](https://www.packer.io/) templates
 
 ***NOTE:**
 
-- Vagrant 2.4.0+ is required for new cpu archetecture support
+- Vagrant 2.4.0+ is required for new cpu architecture support
 - Virutalbox 6.x requires disabling nat config that allows vbox 7.x guests to connect to the host. To use comment out lines #161 and #162 in bento/packer_templates/pkr-variables.pkr.hcl or add variable `vboxmanage = []` to os_pkrvars files.
 - When running packer build command the output directory is relative to the working directory the command is currently running in. Suggest running packer build commands from bento root directory for build working files to be placed in bento/builds/(build_name) directory by default. If the output_directory variable isn't overwritten a directory called builds/(build_name) will be created in the current working directory that you are running the command from
 
@@ -24,6 +24,14 @@ Vagrant.configure("2") do |config|
 end
 ```
 
+### Installing Bento
+
+1. install ruby environment
+1. clone repo
+1. cd <path/to>/bento
+1. gem build bento.gemspec
+1. gem install bento-*.gem
+
 ### Building Boxes
 
 #### Requirements

AMZ_build_virtualbox-ovf.sh → amazonlinux-2-x86_64-virtualbox-build.sh

@@ -12,10 +12,10 @@ IMG="$(wget -q https://cdn.amazonlinux.com/os-images/latest/virtualbox/ -O - | g
 
 # Download vbox vdi
 echo "Downloading Vbox VDI $IMG"
-wget -q -O "$AMZDIR"/amazon.vdi -c https://cdn.amazonlinux.com/os-images/latest/virtualbox/"$IMG"
+wget -q -O "$AMZDIR"/amazon2_x86_64.vdi -c https://cdn.amazonlinux.com/os-images/latest/virtualbox/"$IMG"
 
-if [ ! -f "$AMZDIR"/amazon.vdi ]; then
- echo There must be a file named amazon.vdi in "$AMZDIR"!
+if [ ! -f "$AMZDIR"/amazon2_x86_64.vdi ]; then
+ echo There must be a file named amazon2_x86_64.vdi in "$AMZDIR"!
  echo You can download the vdi file at https://cdn.amazonlinux.com/os-images/latest/virtualbox/
  exit 1
 fi
@@ -36,8 +36,8 @@ echo "Creating the VM"
 # from https://www.perkin.org.uk/posts/create-virtualbox-vm-from-the-command-line.html
 VBoxManage createvm --name $VM --ostype "RedHat_64" --register
 VBoxManage storagectl $VM --name "SATA Controller" --add sata --controller IntelAHCI
-VBoxManage storageattach $VM --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium "$AMZDIR"/amazon.vdi
-VBoxManage storageattach $VM --storagectl "SATA Controller" --port 0 --device 1 --type dvddrive --medium "$AMZDIR"/seed.iso
+VBoxManage storageattach $VM --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium "$AMZDIR"/amazon2_x86_64.vdi
+VBoxManage storageattach $VM --storagectl "SATA Controller" --port 1 --device 0 --type dvddrive --medium "$AMZDIR"/seed.iso
 VBoxManage modifyvm $VM --memory 2048
 VBoxManage modifyvm $VM --cpus 2
 VBoxManage modifyvm $VM --audio-driver none
@@ -48,18 +48,18 @@ echo Sleeping for 120 seconds to let the system boot and cloud-init to run
 VBoxManage startvm $VM --type headless
 sleep 120
 VBoxManage controlvm $VM poweroff --type headless
-VBoxManage storageattach $VM --storagectl "SATA Controller" --port 0 --device 1 --type dvddrive --medium none
+VBoxManage storageattach $VM --storagectl "SATA Controller" --port 1 --device 0 --type dvddrive --medium none
 sleep 5
 
 echo Exporting the VM to an OVF file
-vboxmanage export $VM -o "$AMZDIR"/amazon2.ovf
+vboxmanage export $VM -o "$AMZDIR"/amazon2_x86_64.ovf
 sleep 5
 
 echo Deleting the VM
 vboxmanage unregistervm $VM --delete
 
 echo starting packer build of amazonlinux
-if packer build -timestamp-ui -only=virtualbox-ovf.amazonlinux -var-file="$AMZDIR"/../../os_pkrvars/amazonlinux/amazonlinux-2-x86_64.pkrvars.hcl "$AMZDIR"/../../packer_templates; then
+if packer build -timestamp-ui -var "vbox_source_path=$AMZDIR/amazon2_x86_64.ovf" -var "vbox_checksum=null" -var-file="$AMZDIR"/../../os_pkrvars/amazonlinux/amazonlinux-2-x86_64.pkrvars.hcl "$AMZDIR"/../../packer_templates; then
  echo "Cleaning up files"
  rm -f "$AMZDIR"/*.ovf "$AMZDIR"/*.vmdk "$AMZDIR"/*.iso
 else

amazonlinux-2023-aarch64-virtualbox-build.sh

@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+
+# Getting script directory location
+SCRIPT_RELATIVE_DIR=$(dirname "${BASH_SOURCE[0]}")
+cd "$SCRIPT_RELATIVE_DIR" || exit
+
+# set tmp dir for files
+AMZDIR="$(pwd)/packer_templates/amz_working_files"
+
+echo "Cleaning up old files"
+rm -f "$AMZDIR"/*.iso "$AMZDIR"/*.ovf "$AMZDIR"/*.vmdk "$AMZDIR"/*.vdi
+
+# Get virtualbox vdi file name with latest version number
+IMG="$(wget -q https://cdn.amazonlinux.com/al2023/os-images/latest/kvm-arm64/ -O - | grep ".qcow2" | cut -d "\"" -f 2)"
+
+echo "Downloading $IMG"
+wget -q -O "$AMZDIR"/amazon2023_arm64.qcow2 -c https://cdn.amazonlinux.com/al2023/os-images/latest/kvm-arm64/"$IMG"
+
+echo "Convert qcow2 to vdi"
+qemu-img convert -f qcow2 "$AMZDIR"/amazon2023_arm64.qcow2 -O vdi "$AMZDIR"/amazon2023_arm64.vdi
+
+if [ ! -f "$AMZDIR"/amazon2023_arm64.vdi ]; then
+ echo There must be a file named amazon2023_arm64.vdi in "$AMZDIR"!
+ echo You can download the files at https://cdn.amazonlinux.com/al2023/os-images/latest/
+ exit 1
+fi
+
+echo "Creating ISO"
+hdiutil makehybrid -o "$AMZDIR"/seed.iso -hfs -joliet -iso -default-volume-name cidata "$AMZDIR"/../amz_seed_iso
+
+VM="AmazonLinuxBento"
+echo Powering off and deleting any existing VMs named $VM
+VBoxManage controlvm $VM poweroff --type headless 2> /dev/null
+VBoxManage unregistervm $VM --delete 2> /dev/null
+sleep 5
+
+echo "Creating the VM"
+# from https://www.perkin.org.uk/posts/create-virtualbox-vm-from-the-command-line.html
+VBoxManage createvm --name $VM --ostype "Fedora_64" --register
+VBoxManage storagectl $VM --name "SATA Controller" --add sata --controller IntelAHCI
+VBoxManage storageattach $VM --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium "$AMZDIR"/amazon2023_arm64.vdi
+VBoxManage storageattach $VM --storagectl "SATA Controller" --port 1 --device 0 --type dvddrive --medium "$AMZDIR"/seed.iso
+VBoxManage modifyvm $VM --chipset ich9
+VBoxManage modifyvm $VM --firmware efi
+VBoxManage modifyvm $VM --memory 2048
+VBoxManage modifyvm $VM --cpus 2
+VBoxManage modifyvm $VM --nat-localhostreachable1 on
+VBoxManage modifyvm $VM --vram 33
+VBoxManage modifyvm $VM --graphicscontroller vmsvga
+VBoxManage modifyvm $VM --vrde off
+VBoxManage modifyvm $VM --audio-driver none
+VBoxManage modifyvm $VM --ioapic on
+sleep 5
+
+echo "Sleeping for 120 seconds to let the system boot and cloud-init to run"
+VBoxManage startvm $VM --type headless
+sleep 120
+VBoxManage controlvm $VM poweroff --type headless
+VBoxManage storageattach $VM --storagectl "SATA Controller" --port 1 --device 0 --type dvddrive --medium none
+sleep 5
+
+echo "Exporting the VM to an OVF file"
+vboxmanage export $VM -o "$AMZDIR"/amazon2023_arm64.ovf
+sleep 5
+
+echo "Deleting the VM"
+vboxmanage unregistervm $VM --delete
+
+echo "starting packer build of amazonlinux"
+if packer build -timestamp-ui -var "vbox_source_path=$AMZDIR/amazon2023_arm64.ovf" -var "vbox_checksum=null" -var-file="$AMZDIR"/../../os_pkrvars/amazonlinux/amazonlinux-2023-aarch64.pkrvars.hcl "$AMZDIR"/../../packer_templates; then
+ echo "Cleaning up files"
+ rm -f "$AMZDIR"/*.ovf "$AMZDIR"/*.vmdk "$AMZDIR"/*.iso "$AMZDIR"/*.vdi "$AMZDIR"/*.qcow2
+else
+ exit 1
+fi

amazonlinux-2023-x86_64-virtualbox-build.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+
+# Getting script directory location
+SCRIPT_RELATIVE_DIR=$(dirname "${BASH_SOURCE[0]}")
+cd "$SCRIPT_RELATIVE_DIR" || exit
+
+# set tmp dir for files
+AMZDIR="$(pwd)/packer_templates/amz_working_files"
+
+echo "Cleaning up old files"
+rm -f "$AMZDIR"/*.iso "$AMZDIR"/*.ovf "$AMZDIR"/*.vmdk "$AMZDIR"/*.vdi
+
+# Get virtualbox vdi file name with latest version number
+IMG="$(wget -q https://cdn.amazonlinux.com/al2023/os-images/latest/kvm/ -O - | grep ".qcow2" | cut -d "\"" -f 2)"
+
+echo "Downloading $IMG"
+wget -q -O "$AMZDIR"/amazon2023_x86_64.qcow2 -c https://cdn.amazonlinux.com/al2023/os-images/latest/kvm/"$IMG"
+
+echo "Convert qcow2 to vdi"
+qemu-img convert -f qcow2 "$AMZDIR"/amazon2023_x86_64.qcow2 -O vdi "$AMZDIR"/amazon2023_x86_64.vdi
+
+if [ ! -f "$AMZDIR"/amazon2023_x86_64.vdi ]; then
+ echo There must be a file named amazon2023_x86_64.vdi in "$AMZDIR"!
+ echo You can download the files at https://cdn.amazonlinux.com/al2023/os-images/latest/
+ exit 1
+fi
+
+echo "Creating ISO"
+hdiutil makehybrid -o "$AMZDIR"/seed.iso -hfs -joliet -iso -default-volume-name cidata "$AMZDIR"/../amz_seed_iso
+
+VM="AmazonLinuxBento"
+echo Powering off and deleting any existing VMs named $VM
+VBoxManage controlvm $VM poweroff --type headless 2> /dev/null
+VBoxManage unregistervm $VM --delete 2> /dev/null
+sleep 5
+
+echo "Creating the VM"
+# from https://www.perkin.org.uk/posts/create-virtualbox-vm-from-the-command-line.html
+VBoxManage createvm --name $VM --ostype "Fedora_64" --register
+VBoxManage storagectl $VM --name "SATA Controller" --add sata --controller IntelAHCI
+VBoxManage storageattach $VM --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium "$AMZDIR"/amazon2023_x86_64.vdi
+VBoxManage storageattach $VM --storagectl "SATA Controller" --port 1 --device 0 --type dvddrive --medium "$AMZDIR"/seed.iso
+VBoxManage modifyvm $VM --memory 2048
+VBoxManage modifyvm $VM --cpus 2
+VBoxManage modifyvm $VM --nat-localhostreachable1 on
+VBoxManage modifyvm $VM --vram 10
+VBoxManage modifyvm $VM --graphicscontroller vmsvga
+VBoxManage modifyvm $VM --vrde off
+VBoxManage modifyvm $VM --audio-driver none
+VBoxManage modifyvm $VM --ioapic on
+sleep 5
+
+echo "Sleeping for 120 seconds to let the system boot and cloud-init to run"
+VBoxManage startvm $VM --type headless
+sleep 120
+VBoxManage controlvm $VM poweroff --type headless
+VBoxManage storageattach $VM --storagectl "SATA Controller" --port 1 --device 0 --type dvddrive --medium none
+sleep 5
+
+echo "Exporting the VM to an OVF file"
+vboxmanage export $VM -o "$AMZDIR"/amazon2023_x86_64.ovf
+sleep 5
+
+echo "Deleting the VM"
+vboxmanage unregistervm $VM --delete
+
+echo "starting packer build of amazonlinux"
+if packer build -timestamp-ui -var "vbox_source_path=$AMZDIR/amazon2023_x86_64.ovf" -var "vbox_checksum=null" -var-file="$AMZDIR"/../../os_pkrvars/amazonlinux/amazonlinux-2023-x86_64.pkrvars.hcl "$AMZDIR"/../../packer_templates; then
+ echo "Cleaning up files"
+ rm -f "$AMZDIR"/*.ovf "$AMZDIR"/*.vmdk "$AMZDIR"/*.iso "$AMZDIR"/*.vdi "$AMZDIR"/*.qcow2
+else
+ exit 1
+fi

builds.yml

@@ -6,6 +6,8 @@ default_architectures:
  - 'amd64'
 
 public:
+ - 'amazonlinux-2'
+ - 'amazonlinux-2023'
  - 'almalinux-8'
  - 'almalinux-9'
  - 'amazon-2'

lib/bento/runner.rb

@@ -17,7 +17,7 @@ def initialize(opts)
  @dry_run = opts.dry_run
  @metadata_only = opts.metadata_only
  @debug = opts.debug
- @only = opts.only ||= 'parallels-iso.vm,virtualbox-iso.vm,vmware-iso.vm,qemu.vm'
+ @only = opts.only ||= nil
  @except = opts.except
  @mirror = opts.mirror
  @headed = opts.headed ||= false
@@ -87,7 +87,7 @@ def packer_build_cmd(template, _var_file)
  var_files.each do |var_file|
  cmd.insert(5, "-var-file=#{var_file}") if File.exist?(var_file)
  end if var_files
- cmd.insert(4, "-only=#{only}")
+ cmd.insert(4, "-only=#{only}") if only
  cmd.insert(4, "-except=#{except}") if except
  # Build the command line in the correct order and without spaces as future input for the splat operator.
  cmd.insert(4, "-var cpus=#{cpus}") if cpus

lib/bento/test.rb

@@ -5,12 +5,13 @@
 class TestRunner
  include Common
 
- attr_reader :shared_folder, :boxname, :provider, :box_url, :no_shared, :provisioner
+ attr_reader :shared_folder, :boxname, :provider, :box_url, :no_shared, :provisioner, :errors
 
  def initialize(opts)
  @debug = opts.debug
  @no_shared = opts.no_shared
  @provisioner = opts.provisioner.nil? ? 'shell' : opts.provisioner
+ @errors = []
  end
 
  def start
@@ -23,6 +24,9 @@ def start
  end
  end
  banner("Testing finished in #{duration(time.real)}.")
+ unless errors.empty?
+ raise("Failed Tests:\n #{errors.join("\n")}\n exited #{$CHILD_STATUS}")
+ end
  end
 
  private
@@ -45,8 +49,17 @@ def test_box(md_json)
  temp_dir = "#{bento_dir}/builds/test-kitchen"
  Dir.mkdir(temp_dir) unless Dir.exist?(temp_dir)
  md = box_metadata(md_json)
+ arch = case md['arch']
+ when 'x86_64', 'amd64'
+ 'amd64'
+ when 'aarch64', 'arm64'
+ 'arm64'
+ else
+ raise "Unknown arch #{md_data.inspect}"
+ end
  @boxname = md['name']
  @providers = md['providers']
+ @arch = arch
  @share_disabled = no_shared || /(bsd|opensuse)/.match(boxname) ? true : false
 
  dir = "#{File.expand_path('../../', File.dirname(__FILE__))}/lib/bento/test_templates"
@@ -58,9 +71,14 @@ def test_box(md_json)
 
  Dir.chdir(temp_dir)
  banner("Test kitchen file located in #{temp_dir}")
- test = Mixlib::ShellOut.new('kitchen test', timeout: 900, live_stream: STDOUT)
- test.run_command
- test.error!
+ @providers.each do |k, _v|
+ test = Mixlib::ShellOut.new("kitchen test #{@boxname}-#{@arch}-#{k.tr('_', '-')}", timeout: 900, live_stream: STDOUT)
+ test.run_command
+ if test.error?
+ test.stderr
+ errors << "#{@boxname}-#{@arch}-#{k}"
+ end
+ end
  Dir.chdir(bento_dir)
  FileUtils.rm_rf(temp_dir)
  end

lib/bento/test_templates/kitchen.yml.erb

@@ -4,13 +4,14 @@ provisioner:
 
 platforms:
 <% @providers.each do |k,v| -%>
-- name: <%= @boxname + '-' + k %>
+- name: <%= "#{@boxname}-#{@arch}-#{k}" %>
  driver:
  name: vagrant
  provider: <%= k %>
  vm_hostname: bento-test
  box: bento-<%= @boxname %>
  box_url: file://<%= Dir.pwd %>/builds/<%= v['file'] %>
+ box_arch: <%= @arch %>
  synced_folders:
 <% if k =~ /hyperv/ -%>
  - [".", "/vagrant", "disabled: true"]