SCT mixes up root access on some cases (was: artifacts-ami-test fails with "E: Unable to locate package scylla-manager-agent") #11257
Description
It seems to unrelated with my machine-image patch, artifacts-ami-test keep failing with following error:
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > Command: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=120 -o Dpkg::Options::="--force-confold" -o Dpkg::Options::="--force-confdef" install -y scylla-manager-agent'
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR >
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > Exit code: 100
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR >
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > Stdout:
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR >
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > Reading package lists...
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > Building dependency tree...
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > Reading state information...
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR >
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > Stderr:
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR >
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > E: Unable to locate package scylla-manager-agent
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR >
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR >
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > Traceback (most recent call last):
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/cluster.py", line 4028, in node_setup
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > cl_inst.node_setup(_node, **setup_kwargs)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/cluster.py", line 4947, in node_setup
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > self.install_scylla_manager(node)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/cluster.py", line 4979, in install_scylla_manager
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > node.install_manager_agent(package_path=pkg_path)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/cluster.py", line 1947, in install_manager_agent
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > self.install_package(package_name, package_version)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/utils/decorators.py", line 73, in inner
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > return func(*args, **kwargs)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/cluster.py", line 1930, in install_package
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > self.remoter.sudo(f'{pkg_cmd} install -y {package_name}', ignore_status=ignore_status)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/remote/base.py", line 121, in sudo
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > return self.run(cmd=cmd,
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/remote/remote_base.py", line 651, in run
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > result = _run()
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/utils/decorators.py", line 73, in inner
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > return func(*args, **kwargs)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/remote/remote_base.py", line 642, in _run
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > return self._run_execute(cmd, timeout, ignore_status, verbose, new_session, watchers)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/remote/remote_base.py", line 576, in _run_execute
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > result = connection.run(**command_kwargs)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 622, in run
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > return self._complete_run(channel, exception, timeout_reached, timeout, result, warn, stdout, stderr)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > File "/tmp/jenkins/workspace/releng-testing/artifacts/artifacts-ami-test/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 657, in _complete_run
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > raise UnexpectedExit(result)
14:28:39 < t:2025-06-24 05:28:39,351 f:file_logger.py l:101 c:sdcm.sct_events.file_logger p:ERROR > sdcm.remote.libssh2_client.exceptions.UnexpectedExit: Encountered a bad command exit code!
It occured at: https://jenkins.scylladb.com/job/releng-testing/job/artifacts/job/artifacts-ami-test/70/
And also when I retried: https://jenkins.scylladb.com/job/releng-testing/job/artifacts/job/artifacts-ami-test/95/
I do not get understand how it happened, since I actually able to install scylla-manager-agent on exactly same AMI image, with exactly same scylla-manager repository:
scylla manager repo is: https://s3.amazonaws.com/downloads.scylladb.com/manager/deb/unstable/unified-deb/master/latest/scylla-manager.list
ami id is: ami-0416dd7b2cfec2746
$ sudo curl -L -O https://s3.amazonaws.com/downloads.scylladb.com/manager/deb/unstable/unified-deb/master/latest/scylla-manager.list
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 168 100 168 0 0 2485 0 --:--:-- --:--:-- --:--:-- 2507
$ sudo apt-get update --allow-insecure-repositories
Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble InRelease
Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:4 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:5 https://downloads.scylladb.com/manager/deb/unstable/unified-deb/master/2025-06-10T21:54:47Z stable InRelease [7550 B]
Hit:6 https://downloads.scylladb.com/testing/unstable/scylla/master/deb/unified/next-machine-image/2025-06-17T10:24:29Z/scylladb-master stable InRelease
Get:7 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1155 kB]
Get:8 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [243 kB]
Get:9 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [1250 kB]
Get:10 http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-updates/restricted Translation-en [265 kB]
Get:11 https://downloads.scylladb.com/manager/deb/unstable/unified-deb/master/2025-06-10T21:54:47Z stable/main arm64 Packages [1054 B]
Get:12 https://downloads.scylladb.com/manager/deb/unstable/unified-deb/master/2025-06-10T21:54:47Z stable/main amd64 Packages [1051 B]
Get:13 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [910 kB]
Get:14 http://security.ubuntu.com/ubuntu noble-security/main Translation-en [167 kB]
Get:15 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [860 kB]
Get:16 http://security.ubuntu.com/ubuntu noble-security/universe Translation-en [188 kB]
Get:17 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [1211 kB]
Get:18 http://security.ubuntu.com/ubuntu noble-security/restricted Translation-en [257 kB]
Fetched 6895 kB in 2s (3814 kB/s)
Reading package lists... Done
$ sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=120 -o Dpkg::Options::="--force-confold" -o Dpkg::Options::="--force-confdef" install -y scylla-manager-agent
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
scylla-manager-agent
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 21.8 MB of archives.
After this operation, 51.1 MB of additional disk space will be used.
Get:1 https://downloads.scylladb.com/manager/deb/unstable/unified-deb/master/2025-06-10T21:54:47Z stable/main amd64 scylla-manager-agent amd64 3.6.0~dev-0.20250610.9210892e6-SNAPSHOT [21.8 MB]
Fetched 21.8 MB in 2s (14.2 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package scylla-manager-agent.
(Reading database ... 63537 files and directories currently installed.)
Preparing to unpack .../scylla-manager-agent_3.6.0~dev-0.20250610.9210892e6-SNAPSHOT_amd64.deb ...
scylla:x:108:
scylla:x:106:108::/var/lib/scylla:/usr/sbin/nologin
Unpacking scylla-manager-agent (3.6.0~dev-0.20250610.9210892e6-SNAPSHOT) ...
Setting up scylla-manager-agent (3.6.0~dev-0.20250610.9210892e6-SNAPSHOT) ...
Related with scylladb/scylla-machine-image#712, since it was occured when it tested on next-machine-image.