Running testinfra tests with openstack delegated driver randomly fails

[matmb-cld]

Issue Type

• Bug report

Molecule and Ansible details

ansible --version && molecule --version
ansible 2.9.9
  config file = None
  configured module search path = ['/home/mat/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/mat/Virtualenv/ansible3/lib/python3.8/site-packages/ansible
  executable location = /home/mat/Virtualenv/ansible3/bin/ansible
  python version = 3.8.4rc1 (default, Jul  1 2020, 15:31:45) [GCC 9.3.0]
molecule 3.0.4

Molecule installation method (one of):

• pip

Ansible installation method (one of):

• pip

Detail any linters or test runners used:

Desired Behavior

Running testinfra tests should always return the same status when no change is made between the execution of 2 tests.

Actual Behaviour

When performing molecule verify command:

• the first time all tests passed
• the second time all tests failed du to an SSH issue with the openstack instances.

Firstly: all tests passed

molecule verify
--> Test matrix
    
└── default
    └── verify
    
--> Scenario: 'default'
--> Action: 'verify'
--> Executing Testinfra tests found in /home/mat/GIT/ansible_project/ansible_role/xymon-client/molecule/default/tests/...
    ============================= test session starts ==============================
    platform linux -- Python 3.8.4rc1, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
    rootdir: /home/mat/GIT/ansible_project/ansible_role/xymon-client/molecule/default
    plugins: testinfra-5.2.1
collected 8 items                                                              
    
    tests/test_default.py ..                                                 [ 25%]
    tests/test_common.py ......                                              [100%]
    
    ============================== 8 passed in 15.70s ==============================
Verifier completed successfully.

Secondly: all tests failed:

When retrying to perform molecule verify* command 5 seconds after the first try, all tests failed with the error:

RuntimeError: CommandResult(command=b'test -e /etc/default/xymon-client', exit_status=255, stdout=None, stderr=b'kex_exchange_identification: read: Connection reset by peer\r\n')

molecule verify
--> Test matrix
    
└── default
    └── verify
    
--> Scenario: 'default'
--> Action: 'verify'
--> Executing Testinfra tests found in /home/mat/GIT/ansible_project/ansible_role/xymon-client/molecule/default/tests/...
    ============================= test session starts ==============================
    platform linux -- Python 3.8.4rc1, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
    rootdir: /home/mat/GIT/ansible_project/ansible_role/xymon-client/molecule/default
    plugins: testinfra-5.2.1
collected 8 items                                                              
    
    tests/test_default.py FF                                                 [ 25%]
    tests/test_common.py FFFFFF                                              [100%]
    
    =================================== FAILURES ===================================
    _____________ test_xymon_client_default[ansible://xymon-debian10] ______________
    
    host = <testinfra.host.Host ansible://xymon-debian10>
    
        def test_xymon_client_default(host):
            f = host.file('/etc/default/xymon-client')
            ansible_vars = host.ansible.get_variables()
            hostname = ansible_vars['inventory_hostname']
    >       assert f.contains('^XYMONSERVERS="127.0.0.1"$')
    
    tests/test_default.py:13:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    /home/mat/Virtualenv/ansible3/lib/python3.8/site-packages/testinfra/modules/file.py:118: in contains
        return self.run_test("grep -qs -- %s %s", pattern, self.path).rc == 0
    /home/mat/Virtualenv/ansible3/lib/python3.8/site-packages/testinfra/host.py:94: in run_test
        return self.run_expect([0, 1], command, *args, **kwargs)
    /home/mat/Virtualenv/ansible3/lib/python3.8/site-packages/testinfra/host.py:75: in run
        return self.backend.run(command, *args, **kwargs)
    /home/mat/Virtualenv/ansible3/lib/python3.8/site-packages/testinfra/backend/ansible.py:47: in run
        return host.run(command)
    /home/mat/Virtualenv/ansible3/lib/python3.8/site-packages/testinfra/host.py:75: in run
        return self.backend.run(command, *args, **kwargs)
    /home/mat/Virtualenv/ansible3/lib/python3.8/site-packages/testinfra/backend/ssh.py:34: in run
        return self.run_ssh(self.get_command(command, *args))
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    self = <testinfra.backend.ssh.SshBackend object at 0x7f908138c310>
    command = 'grep -qs -- \'^XYMONSERVERS="127.0.0.1"$\' /etc/default/xymon-client'
    
        def run_ssh(self, command):
            cmd, cmd_args = self._build_ssh_command(command)
            out = self.run_local(
                " ".join(cmd), *cmd_args)
            out.command = self.encode(command)
            if out.rc == 255:
                # ssh exits with the exit status of the remote command or with 255
                # if an error occurred.
    >           raise RuntimeError(out)
    E           RuntimeError: CommandResult(command=b'grep -qs -- \'^XYMONSERVERS="127.0.0.1"$\' /etc/default/xymon-client', exit_status=255, stdout=None, stderr=b'kex_exchange_identification: read: Connection reset by peer\r\n')
    
    /home/mat/Virtualenv/ansible3/lib/python3.8/site-packages/testinfra/backend/ssh.py:77: RuntimeError
    ______________ test_xymon_client_default[ansible://xymon-debian9] ______________
.....

I've rewritten the test in ansible instead of testinfra and with ansible verifier, there is no SSH issue.
So I guess something weird happens between testinfra and SSH when running molecule verify command.

Additionnal info

molecule.yml

dependency:
  name: galaxy
driver:
  name: delegated
  security_group: xymon-default-molecule
  network: molecule
  ssh_user: admin
  keypair: xymon-default
lint: |
  set -e
  yamllint .
  ansible-lint -x 503,303 molecule/default/*.yml
platforms:
  - name: xymon-debian10
    image: Debian_10
    flavor: clouds.small
  - name: xymon-debian9
    image: Debian_9
    flavor: clouds.small
provisioner:
  name: ansible
  config_options:
    defaults:
      ansible_managed: 'Molecule Tests'
verifier:
  name: testinfra

converge.yml

- name: Converge
  hosts: all
  become: true
  vars:
    ansible_python_interpreter: auto_silent
  pre_tasks:
    - name: update apt cache
      apt:
        update_cache: true
      changed_when: false
  roles:
    - role: xymon-client

testinfra

test_common.py

import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')


def test_package(host):
    p = host.package('xymon-client')
    assert p.is_installed


def test_service(host):
    s = host.service('xymon-client')
    assert s.is_running
    assert s.is_enabled


def test_xymon_client_default(host):
    f = host.file('/etc/default/xymon-client')
    assert f.exists
    assert f.is_file
    assert f.user == 'root'
    assert f.group == 'root'
    assert f.mode == 0o644

test_default.py

import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')


def test_xymon_client_default(host):
    f = host.file('/etc/default/xymon-client')
    ansible_vars = host.ansible.get_variables()
    hostname = ansible_vars['inventory_hostname']
    assert f.contains('^XYMONSERVERS="127.0.0.1"$')
    assert f.contains('^CLIENTHOSTNAME="%s"$' % hostname)

[nikParasyr]

Hello @matmb-cld .

So I guess something weird happens between testinfra and SSH when running molecule verify command.

I'm wondering whether this is a molecule-openstack issue or a testinfra issue. Reason is that when changing to ansible verify it works as expected. With a quick look I can see that maybe this issue: pytest-dev/pytest-testinfra#528 is related.

In any case I'll have a look later this week to troubleshoot this a bit. Try a couple more drivers (e.g aws) to see what happens

[berendt]

We now have this problem as well. The testinfra tests fail randomly.

[palchyk-alex]

Issue still occurs.