Playbook fails with ImportError: No module named http.client

[m-kratochvil]

COMPONENT NAME

f5networks.f5_modules.bigip_lx_package

Environment

ANSIBLE VERSION

ansible [core 2.15.0]
  config file = /home/d071955/repos/ccloud-net/ansible.cfg
  configured module search path = ['/home/user/repos/ccloud-net/plugins/modules']
  ansible python module location = /home/user/.venvs/ccloudnet/lib/python3.10/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/user/.venvs/ccloudnet/bin/ansible
  python version = 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] (/home/user/.venvs/ccloudnet/bin/python3)
  jinja version = 3.1.2
  libyaml = True

ansible-galaxy collection list | grep f5_modules
f5networks.f5_modules 1.24.0

BIGIP VERSION

Sys::Version
Main Package
  Product     BIG-IP
  Version     17.1.0.1
  Build       0.0.4
  Edition     Point Release 1
  Date        Fri Apr  7 07:24:51 PDT 2023

OS / ENVIRONMENT

Ubuntu 22.04 VM
Python 3.10.6

SUMMARY

Playbook to install AS3 plugin used to work, suddenly it fails. Other playbooks/modules might be affected, I only ran into this issue with the particular module specified below.

STEPS TO REPRODUCE

Run a playbook to install an AS3 package, using following module:

    - name: Install AS3 package
      f5networks.f5_modules.bigip_lx_package:
        provider: "{{ provider }}"
        package: "/var/config/rest/downloads/{{ filename }}"
        retain_package_file: yes

EXPECTED RESULTS

Playbook should run and complete without failure, installing the AS3 package.

ACTUAL RESULTS

The playbook failed with following error:

<10.0.0.1> (1, b'Traceback (most recent call last):\r\n  File "/root/.ansible/tmp/ansible-tmp-1694773488.3828719-230249-242541842573464/AnsiballZ_bigip_lx_package.py", line 107, in <module>\r\n    _ansiballz_main()\r\n  File "/root/.ansible/tmp/ansible-tmp-1694773488.3828719-230249-242541842573464/AnsiballZ_bigip_lx_package.py", line 99, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File "/root/.ansible/tmp/ansible-tmp-1694773488.3828719-230249-242541842573464/AnsiballZ_bigip_lx_package.py", line 48, in invoke_module\r\n    run_name=\'__main__\', alter_sys=True)\r\n  File "/usr/lib/python2.7/runpy.py", line 176, in run_module\r\n    fname, loader, pkg_name)\r\n  File "/usr/lib/python2.7/runpy.py", line 82, in _run_module_code\r\n    mod_name, mod_fname, mod_loader, pkg_name)\r\n  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code\r\n    exec code in run_globals\r\n  File "/tmp/ansible_f5networks.f5_modules.bigip_lx_package_payload_l_9W6O/ansible_f5networks.f5_modules.bigip_lx_package_payload.zip/ansible_collections/f5networks/f5_modules/plugins/modules/bigip_lx_package.py", line 130, in <module>\r\n  File "/tmp/ansible_f5networks.f5_modules.bigip_lx_package_payload_l_9W6O/ansible_f5networks.f5_modules.bigip_lx_package_payload.zip/ansible_collections/f5networks/f5_modules/plugins/module_utils/teem.py", line 20, in <module>\r\nImportError: No module named http.client\r\n', b'Shared connection to 10.0.0.1 closed.\r\n')

To resolve on short-term, I went into the following file:

.ansible/collections/ansible_collections/f5networks/f5_modules/plugins/module_utils/teem.py

and removed references to http.client

Then the playbook failed with different error:

<10.0.0.1> (1, b'\r\n{"exception": "Traceback (most recent call last):\\n  File \\"/tmp/ansible_f5networks.f5_modules.bigip_lx_package_payload_dsi9fgwp/ansible_f5networks.f5_modules.bigip_lx_package_payload.zip/ansible_collections/f5networks/f5_modules/plugins/modules/bigip_lx_package.py\\", line 109, in <module>\\nModuleNotFoundError: No module named \'packaging\'\\n", "failed": true, "msg": "Failed to import the required Python library (packaging) on qa-de-1-lb011a-gtm.cc.qa-de-1.cloud.sap\'s Python /usr/bin/python3.8. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter", "invocation": {"module_args": {"provider": {"server": "10.46.100.214", "server_port": 443, "user": "admin", "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "no_f5_teem": true, "validate_certs": false, "transport": "rest", "timeout": null, "auth_provider": null}, "package": "/var/config/rest/downloads/f5-appsvcs-3.36.1-1.noarch.rpm", "retain_package_file": true, "state": "present"}}}\r\n', b'Shared connection to 10.0.0.1 closed.\r\n')

I went into the following file:

.ansible/collections/ansible_collections/f5networks/f5_modules/plugins/modules/bigip_lx_package.py

and removed references to packaging.version

After that the playbook executed correctly.

[pgouband]

Hi @m-kratochvil,

I tested the following playbook with f5networks.f5_modules version 1.26.0 on device using BIG-IP v17.1 with success.

- hosts: all
  collections:
    - f5networks.f5_modules
  connection: local

  vars:
    provider:
      server: "X.X.X.X"
      user: "admin"
      password: "mysecretpassword"
      server_port: 443
      validate_certs: no
      no_f5_teem: yes

  tasks:
     - name: Install AS3
       f5networks.f5_modules.bigip_lx_package:
         package: f5-appsvcs-3.47.0-8.noarch.rpm
         provider: "{{ provider }}"
       delegate_to: localhost

[m-kratochvil]

Hi @pgouband thanks for the feedback. I believe it's working for you because you are using the connection: local and delegate_to: localhost in your playbook. I don't use these options, I execute the module on the Big-IP itself. Can you try the same and see if it works that way for you? I noticed you are using a bit newer collection version, I'll see if I can upgrade and re-test.

[pgouband]

Hi @m-kratochvil,

Can you check you are using python3 because the error looks like python2 is used?

[m-kratochvil]

I am using python 3.10.6 on the ansible controller. I initially also thought it may be an issue with python2 because that is what runs on the Big-IP but as this module was executing fine recently against TMOS version 15.1.3, it should not be a problem against TMOS version 17.1.0.1 where the same python version (2.7) is used.

[jussisjostrom]

I think the problem is indeed that you try to run the task on the BIGIP itself. The python on the BIGIP is quite limited, and is probably missing the http.client. Easiest way to fix the issue is to add the delegate_to: localhost to the task, as adding python modules to BIGIP is not a good idea. I had the same issue on bigip_routedomain module because I forgot to add delegate_to...
Also, all example tasks on https://docs.ansible.com/ansible/latest/collections/f5networks/f5_modules/bigip_lx_package_module.html use the delegate_to: localhost.

[pgouband]

Hi @m-kratochvil,

Can you test adding delegate_to?

When you said "I execute the module on the Big-IP itself", that means you have installed ansible on the device?

[m-kratochvil]

Appreciate your feedback @jussisjostrom @pgouband

When you said "I execute the module on the Big-IP itself", that means you have installed ansible on the device?

No, that was just poor choice of words, I meant I execute the module without delegate_to

If I add delegate_to: localhost, the module fails with "The specified LX package was not found at because the module is looking for the rpm package on the Ansible controller.

I understand your suggestion to run with delegate_to, since it is the standard way to run the imperative f5 modules.
But the bigip_lx_package is an exception. I asked for the option to run it without delegate_to back in 2020, it was implemented via #1717 and was working fine, without any issue, since 2020 until now.

The obvious question is what has recently changed - and nothing comes to my mind, we run same ansible version, same python version on the controller, same f5 collection version. The playbook fails against both our TMOS versions 15.1.3 and 17.1.0.1
It may be worth mentioning that there was already an issue (#2176) with running this module without delegate_to which was resolved by adding no_f5_teem: True to the provider dict.

I am really out of any ideas why this has suddenly started to fail.

[pgouband]

Hi @m-kratochvil,

Adding delegate_to is impossible in your environment?

[m-kratochvil]

@pgouband It is technically possible but we really want to avoid that. Reason is that we currently store the AS3 rpm packages in central, controlled storage and each Big-IP downloads it from there during installation.
If we use delegate_to, every user will have to download the package to their local ansible environment, which we wanted to avoid, for I think obvious reasons..

[pgouband]

Hi @m-kratochvil,

The answer is to use delegate_to and the file needs to be on the ansible client.
What was actually implemented for the 1717 is the retain_package_file option which specifies whether the package file should be deleted or not after successful installation.

[m-kratochvil]

Hi @pgouband the fix/feature for Issue 1717 was implemented via 7257354 and if you look in the diff at a file test/integration/targets/bigip_lx_package/tasks/issue-01717.yaml, you can clearly see that the bigip_lx_package module is used without delegate_to. I believe it works via the function check_file_exists_on_device.
This is exactly the way we used the module, and again, it worked fine until around Sept 2023.

The answer is to use delegate_to and the file needs to be on the ansible client.

That is for sure one option to install AS3 with Ansible, and then there is the other option, without having to upload the AS3 package from the ansible client, and that's what I am after, as it is much better fit to our environment. We are a team of 20 people, managing well over 500 Big-IPs, it is way more practical and safe for use to use central storage for such files.

I hope it clarifies things, and thanks for looking into it.

[urohit011]

Hi @m-kratochvil , the test/integration/targets/bigip_lx_package/tasks/issue-01717.yaml is part of a role for which the entry point is test/integration/bigip_lx_package.yaml, there you can see connection: local on line 24, which does the same job as delegate_to.

[m-kratochvil]

Hi @urohit011 thanks for the hint, I have missed that. But I can also see that in test/integration/targets/bigip_lx_package/tasks/issue-01717.yaml the AS3 package is fetched from GitHub, which means the sub-sequent bigip_lx_package module is not supposed to look for it on the ansible controller but instead it should look for it in the target Big-IP's local /var/tmp/ directory.

If connection: local is the same as delegate_to, then the module will search for the package on the ansible controller.
I'm not sure I understand then how this integration test is working, or perhaps connection: local is not the same as delegate_to..?

Anyway, I stress again, it was working for us fine for several years, the question is why it suddenly stopped. Plus, it actually still works without the connection: local or delegate_to, if I make the adjustments described in my initial post.

I realized I didn't share the actual playbook we have used, so here it is:

Role entry point:

- name: AS3 package management.
  hosts: bigips
  gather_facts: false
  vars_files:
    - vault
    - global_vars

  tasks:
    
    - name: Call the role to manage the AS3 package.
      ansible.builtin.include_role:
        name: as3

Role as3 - tasks/main.yaml:

- name: Download and install AS3.
  block:

    - name: Download the AS3 package from Swift object storage.
      ansible.builtin.get_url:
        backup: no
        dest: /var/config/rest/downloads/
        force: no
        url: "https://storage.swift.abc/loadbalancer/f5-appsvcs-3.5.0-3.noarch.rpm"
        use_proxy: no
        validate_certs: no

    - name: Install the downloaded AS3 package.
      f5networks.f5_modules.bigip_lx_package:
        provider: "{{ provider }}"
        package: "/var/config/rest/downloads/f5-appsvcs-3.5.0-3.noarch.rpm"
        retain_package_file: yes

[urohit011]

In the tasks/main.yaml you just shared above the first task is doing the job of downloading the AS3 package from Swift storage to the controller and it is using the built-in ansible module get_url. So the bigip_lx_package never does the job of transferring RPM package from swift storage to BIGIP box. Second thing is the modulenotfounderror could be fixed if we explicitly set the ansible_pathon_interpreter to python3 to make sure you're not using python2.

[m-kratochvil]

@urohit011 I respectfully disagree. The get_url module downloads the file from Swift to the Big-IP, not to my ansible controller (for that I would have to use connection: local or delegate_to: localhost with the task). This is my understanding of how the ansible modules work, and for get_url module, even the docu suggest this:

The remote server must have direct access to the remote resource.

Regarding ansible_python_interpreter, I have tried that. Since there is no connection: local or delegate_to: localhost used, ansible always uses python 2.7 as it executes on the Big-IP, and that is the default python version there.
As far as my ansible controller, I don't have python 2 installed (but that should not be important anyway in context of this issue).

[urohit011]

@m-kratochvil , yes I agree that get_url will download the rpm on the bigip box, I was wrong earlier. The workaround I can suggest is use get_url to download the rpm on the controller and and use bigip_lx_package to upload it and add another task to delete the rpm from the controller. The reason I suggest this is, bigip_lx_package should run with local or localhost, if it runs on the box you'll get modulenotfound error for the packaging module.

[m-kratochvil]

I have tested my original playbook with various older version of the f5networks.f5_modules collection and I can confirm that collection version 1.22.0 is where the failure was introduced. On collection versions prior to that the playbook works fine.

It would perhaps suggest that support for python2.7 was dropped in that collection version, can anybody confirm? The release notes of collection version 1.22.0 do not list any such change.

For now, I downgraded to collection version 1.21.0 and everything is working fine for me. I admit I could've thought of this test earlier..

[barakbd]

Same issue here when upgrading to 1.25.1