Netbox as an inventory source.

[kollross]

I'm trying to configure semaphore to use a dynamic inventory from Netbox (much like you can do via ansible on the CLI).

Is this feasible?

I tried to dump the current ansible inventory file (backhanded by netbox) and dumping it into a yml file (ansible-inventory --list > inventory.yml) and specifying that as a "inventory file" but that didn't appear to work correctly.

[ansibleguy]

Greetings.

I cannot reproduce your use-case as I don't have a netbox set-up, but as I see it - as far as all the needed inventory files exist on the semaphore-server you should be able to use it with the 'inventory file' option.
We currently use this inventory-type with a local clone of our repository (so we don't have to clone it every time a role runs..) and it works like a charm.

Have you tried running the Ansible CLI with the inventory-files you generated?
That might help you debug it.

- AnsibleGuy

[kollross]

Unfortunately that doesn't work. I used the ansible-inventory --list output to a file, but it semaphore wasn't able to load that as a list. Given that ansible can natively use netbox as an inventory, it kinda surprises me semaphore can't do the same.

[ansibleguy]

Hmm.
Have you tried adding the dynamic-inventory script as inventory-file inside Semaphore?
From what I see in the codebase it is passed 1-to-1 as 'inventory' argument to ansible-playbook.

	var inventory string
	switch t.inventory.Type {
	case db.InventoryFile:
		inventory = t.inventory.Inventory
       ...
	args = []string{
		"-i", inventory,
	}

I haven't used dynamic inventories, but from the ansible docs it seems this is all there is to it, isn't it?

ansible -i openstack_inventory.py all -m ansible.builtin.ping

[kollross]

I don't pass an inventory to ansible when running playbooks. It simply uses netbox as the backend inventory by default.

Example: ansible-playbook something.yml

Inside of the playbook I will have something like:

hosts: device_type_network_device

This calls a group in netbox called network_device that the playbook runs against.

[ansibleguy]

Interesting to know - I did not yet come around to play around with dynamic inventories.

I am pretty sure this is doable as it is an existing feature of the basic Ansible installation that is used by semaphore.

I've read into it a little - you could try the following:

• install the netbox collection in the context of the user that runs semaphore (I guess you have done that already)
• add a generic 'netbox_inventory.yml' (or however you want to call it) on the semaphore host
• in this file you configure your netbox plugin - as described in the netbox docs

  ---
  plugin: netbox.netbox.nb_inventory
  api_endpoint: http://AAAA:8000
  validate_certs: true
  config_context: false
  ...

• after that add the 'ANSIBLE_INVENTORY' environmental variable to your semaphore task 'environment'
  Its value should be the 'netbox_inventory.yml' (or whatever) with its absolute path

Information source: netboxlabs

What could be still a problem: I think you need to select an inventory in the 'task-template' Web-UI form of Semaphore.
Maybe you are better of adding the 'netbox_inventory.yml' as inventory inside the WebUI

Edit: one can set the 'ANSIBLE_INVENTORY' as environmental variable instead of using the 'ansible.cfg'

[ansibleguy]

Also: Related to #1236

[kollross]

In my case semaphore is running on the same system I have ansible setup, so the first 3 bullet points were already completed.

Environment -> Environment variables: I set this:
{ "ANSIBLE_INVENTORY": "/etc/ansible/netbox_inventory.yml" }

Like you mentioned, when creating a task, I need to specify an inventory, so I created a new inventory, and specific a static yaml and pasted in the contents of "netbox_inventory.yml". Then I created a normal task, but it failed with the following output (truncated)

Run TaskRunner with template: Backup Network Devices
10:31:06 AM
[WARNING]: * Failed to parse /home/semaphore/inventory_10.yml with auto
10:31:06 AM
plugin:
10:31:06 AM
netbox-base-path="" data-netbox-color-mode="unset" >
10:31:06 AM
10:31:06 AM
content="initial-scale=1, maximum-scale=1, user-scalable=no, width=device-
10:31:06 AM
width, viewport-fit=cover" />
10:31:06 AM
NetBox
10:31:06 AM
src="/static/setmode.js" onerror="window.location='/media-
10:31:06 AM
failure/?filename=setmode.js'">
10:31:06 AM
type="text/javascript"> (function () { initMode() })();
10:31:06 AM
window.CSRF_TOKEN =

So it looks like it's trying to put the html from the netdot server (vs a json output). I changed the inventory from a static yaml to file and gave it the full path to the netbox_inventory.yml file, but got similar results.

[craigarms]

Hi all,

I'm using Semaphore in a Docker env and got it working with Netbox no issues, (Except getting the Survey Variable to take the ansible_limit, I have yet to debug this)

But in essence Semaphore doesn't care if your Inventory file is a real Json inventory or the config to access Netbox, as @ansibleguy said create you inventory.yml as per the Netbox docs and feed that to Semaphore which will in turn pass it straight to Ansible.

Any errors you get will be Ansible pure errors :

• At the root of your repo have the requirements file :

collections:
  - netbox.netbox

• Also you'll need to make sure you have pytz installed through pip so that it works correctly

As I said i'm in a Docker env so I created a new container with my requirements:

FROM semaphoreui/semaphore:latest

USER root
RUN apk add --no-cache python3 py3-pip
RUN pip install pywinrm jmespath netaddr passlib requests pytz matrix_client pynetbox

USER semaphore

I hope this help, I like this project :)

[kollross]

@craigarms still doesn't work.

This is what my semaphore inventory config looks like.

The inventory file itself looks like

plugin: netbox.netbox.nb_inventory
api_endpoint: https://netbox.mydomain.com
token: hidden
config_context: True
validate_certs: False
group_by:
  - device_roles
  - platforms
  - tags
query_filters:
  - status: "active"
  - has_primary_ip: 'true'
compose:
  ansible_network_os: platform.slug

Then running the task in semaphore I get the following:

Already up to date.
10:12:47 PM
Get current commit hash
10:12:47 PM
Get current commit message
10:12:47 PM
No collections/requirements.yml file found. Skip galaxy install process.
10:12:47 PM
No roles/requirements.yml file found. Skip galaxy install process.
10:12:50 PM
Started: 14
10:12:50 PM
Run TaskRunner with template: Backup Network Devices
10:12:50 PM
[WARNING]: * Failed to parse /etc/ansible/netbox_inventory.yml with auto
10:12:50 PM
plugin:
10:12:50 PM
netbox-base-path="" data-netbox-color-mode="unset" >
10:12:50 PM
10:12:50 PM
content="initial-scale=1, maximum-scale=1, user-scalable=no, width=device-
10:12:50 PM
width, viewport-fit=cover" />
10:12:50 PM
NetBox
10:12:50 PM
src="/static/setmode.js" onerror="window.location='/media-
10:12:50 PM
failure/?filename=setmode.js'">
10:12:50 PM
type="text/javascript"> (function () { initMode() })();
10:12:50 PM
window.CSRF_TOKEN =
10:12:50 PM
"dAgLFRIO9w2ORT7H5OdNzDo4DiOBXcfbwMwljKox2F0nzm3shQ4pT96rwaajelp4";
10:12:50 PM
10:12:50 PM
external.css?v=3.5.8" onerror="window.location='/media-
10:12:50 PM
failure/?filename=netbox-external.css'" />
10:12:50 PM
href="/static/netbox-light.css?v=3.5.8" onerror="window.location='/media-
10:12:50 PM
failure/?filename=netbox-light.css'" />
10:12:50 PM
href="/static/netbox-dark.css?v=3.5.8" onerror="window.location='/media-
10:12:50 PM
failure/?filename=netbox-dark.css'" />
10:12:50 PM
media="print" href="/static/netbox-print.css?v=3.5.8"
10:12:50 PM
onerror="window.location='/media-failure/?filename=netbox-print.css'" />

[craigarms]

Hi,

Yes I realised i wasn't clear/made a mistake, the collections requirements file needs to be in the same folder as your playbook under collections/requirements.yml

There is an issue currently open to be able to have this at the root also.

Even though your task insn't reading the requirements file it doesn't seem to be your issue here, i had a similar issue when I had a version mismatch between the Netbox API and the netbox.netbox collection.

There seems to be 2 versions that Ansible is downloading :

netbox.netbox 3.7.1 is in my global collections

netbox.netbox 3.13.0 is in my user local collections

If your not in a Docker env you should try ansible-galaxy collection install netbox.netbox to get the latest version, then do an ansible-galaxy collection list to check if the latest version is installed to a path that the Ansible in Semaphore will look in.

If you are in the Docker Semaphore, changing the directory of your requirements.yml should do the trick

[kollross]

FWIW, I did add a "requirements.yaml" file into the repo with what you stated above, but doesn't appear to do anything.

[imp1sh]

So is it possible to have a dynamic netbox inventory or not? I don't get it.
The requirements.txt and/or requirements.yml should be in the git repo, right?
In the docs it says something about putting it into /home/semaphore which seems like a bad idea to me.

[craigarms]

Yes,
Mine is working fine, the issues described above are pure-Ansible / Plugin issues not with Semaphore.

Semaphore doesn't manipulate the inventory, just passes it to Ansible, so any valid Ansible dynamic inventory plugin will work given that you have the correct dependancies.

The requirements.yml should be in your repo in the playbook dir under collections/requirements.yml

Then at each run it will check the requirements are satisfied:

5:54:49 PM
Task 59 added to queue
5:54:51 PM
Started: 59
5:54:51 PM
Run TaskRunner with template: Onboard new server
5:54:51 PM
Preparing: 59
5:54:51 PM
Cloning Repository https://gitlab/my.git
5:54:51 PM
Cloning into 'repository_1_2'...
5:54:57 PM
Starting galaxy collection install process
5:54:57 PM
Process install dependency map
5:55:27 PM
Starting collection install process
5:55:27 PM
Downloading https://galaxy.ansible.com/download/netbox-netbox-3.14.0.tar.gz to /tmp/semaphore/.ansible/tmp/ansible-local-86is2ngce0/tmp8t0zxmio/netbox-netbox-3.14.0-jrvxmmsl
5:55:28 PM
Installing 'netbox.netbox:3.14.0' to '/tmp/semaphore/.ansible/collections/ansible_collections/netbox/netbox'
5:55:29 PM
netbox.netbox:3.14.0 was installed successfully

[imp1sh]

OK, but where put I the requirements.txt (pip) and the requirements.yml (galxy roles / collections)?

[kollross]

In the root of your git repo, create a directory called collections, and put the yank file inside that directory.

[craigarms]

The documentation needs clarifying, when i get time i may create a few documentation related PRs

• the role requirements file needs to be in the root of your repo under roles/requirements.yml
• the collection requirements file needs to be in the playbook directory under collections/requirements.yml
• pip requirements are not implemented at run time see PR Enhancement: Installation of User Python Dependencies at Container Runtime #1250 you need to build your container with them it it of install them yourself of your not in a container see the discussion Adding lib hcloud does not help to run playbook: Failed to import the required Python library #1405 for an example of Dockerfile

So if like me you have playbooks in a subdir of your repo you'd have

Repo root
 |- inventory.yml
 |- playbooks/
 |   | - myplaybook.yml
 |   | - collections/
 |       | - requirements.yml
 | - roles/
      | - requirements.yml

[imp1sh]

That is certainly very helpful, thank you so much @craigarms

Edit:
One more thing. I will be fine so far to create my own image, but in the long term this might get some consideration. Every environment would need their very own kind of pip requirements. Neither of the following would not be the best approach.

• running multiple instances of semaphore containers each with their own pip modules nor
• having one image with all dependencies for all the projects / environments

[craigarms]

Glad I could help,

There is the PR mentioned to have the dependencies added at runtime, I've also seen an issue suggesting running plays in a venv which would address issues of dependency conflicts.

Personally for now I have a requiements.txt file added to my build and pip install -r requirements.txt for the build I plan on having a cron which rebuilds the image each night and docker compose up of the Semaphore project.
If nothing changes nothing should happen, as long as i don't forget to pin the different versions :)

[imp1sh]

There seems to be a problem with the collection requirements.yml. It can't be found, even if it's there.
See also: #1144

[craigarms]

I needs to be in the same dir as your playbooks, in a dir called collections and the file needs to be requirements.yml

The issue is that semaphore doesn't look anywhere else

[tboerger]

Looks like you got Netbox working so far, maybe you can provide some instructions within the documentation at https://github.com/semaphoreui/ansible-semaphore-docs?

[mariiko-dev]

While addressing my issue, I decided to try using a plugin that's built into Ansible and is also integrated into Semaphore: netbox.netbox.nb_inventory. If you're interested in setting up NetBox integration with Semaphore, here's the simplest guide: https://github.com/mariiko-dev/NetBox-Semaphore-Integration.