Oxidized Local Saved Copies Are Not Updating

[CootieCooter]

Hey all, so we are running Oxidized and Oxidized Web and it has come to my attention that our local copies of device configs saved at /home/oxidized/configs are not updating with newer copies. In fact, it shows a lot of them were last updated July 29, 2024, while it is November 7, 2024. However, on Oxidized Web, it shows the successful polled devices were last updated November 7, 2024. I have tampered around with the config file and have yet to figure out why the local copies are not updating. This is very essential for us for redundancy.

Here's what we have for that in the config file -->

output:
  default: git
  file:
    directory: "/home/oxidized/configs"

[robertcheramy]

Can't reproduce. Works for me.

[CootieCooter]

@robertcheramy

Weird, we are still running into this issue. Anything else I can add to this issue case to provide more info??

[robertcheramy]

Have a look at https://github.com/ytti/oxidized/blob/master/docs/Issues.md and give more details (including your full config and used versions)

[CootieCooter]

Server hosting Oxidized version: Ubuntu 20.04.6 LTS (Focal Fossa).
Oxidized and Ruby versions: 0.30.1 & ruby 3.2.3

----- The FULL Oxidized config -----

<me>@<Oxi Server>:/home/oxidized/.config/oxidized$ cat config
---
#----Directed Oxidized to the Models directory, used for troubleshooting ----
#model_dir: /home/oxidized/.config/oxidized/lib/oxidized/model
#model_dir: /home/oxidized/.config/oxidized/model/

#----Global Credentials. Commented out to use group/model-specific credentials------


#-----Models, no need to re-configure-----------
model: comware
#model: Comware_qa
#model: h3c
model: panos
model: fsos
model: fs
model: fsnsg
model: cnos

#----Additional Settings, no need to re-configure------
resolve_dns: true
interval: 300
use_syslog: false
log: /home/oxidized/.config/oxidized/oxidized_process.log
debug: false
threads: 30
timeout: 30
retries: 3
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
#rest: is the Oxidized Web rest api. Very helpful.
rest: 127.0.0.1:8888
next_adds_job: false
vars:
  auth_methods: [ "none", "publickey", "password", "keyboard-interactive" ]

#-------Groups, no need to re-configure---------
groups:
  comware:
    username: oxidized
    password: <PasswordA>
    model: comware
  Comware_qa:
    username: qa.oxidized
    password: <PasswordB>
    model: Comware_qa
  hpe:
    username: oxidized
    password: <PasswordA>
    model: comware
  hpe_qa:
    username: qa.oxidized
    password: <PasswordB>
    model: comware_qa
  panos:
    username: oxidized
    password: <PasswordA>
    model: panos
  panos_qa:
    username: qa.oxidized
    password: <PasswordB>
    model: panos
  #tmos:
    #username:
    #password:
  fs:
    username: oxidized
    password: <PasswordA>
    model: fs
  fs_qa:
    username: qa.oxidized
    password: <PasswordB>
    model: fs_qa

#-------------Models section, no need to re-configure-------------
#models: {}
models:
  comware:
    username: oxidized
    password: <PasswordA>
  Comware_qa:
    username: qa.oxidized
    password: <PasswordB>
  panos:
    username: oxidized
    password: <PasswordA>
  fs:
    username: oxidized
    password: <PasswordA>
  fs_qa:
    username: qa.oxidized
    password: <PasswordB>


#------Additional Log/File Settings, No need to re-configure-----
pid: "/home/oxidized/.config/oxidized/pid"
crash:
  directory: "/home/oxidized/.config/oxidized/crashes"
  hostnames: false
stats:
  history_size: 10
input:
  default: ssh
  debug: false
  ssh:
    secure: true
  utf8_encoded: true
output:
  default: git
  file:
    directory: "/home/oxidized/configs"
  git:
    user: oxidized
    email: <users email>
    repo: "/home/oxidized/.config/oxidized/DeviceConfigs.git"
    enabled: true
    auto_commit: true
 

#------Source(s) for Oxidized to gather data from, no need to re-configure-----
source:
  default: csv
  csv:
    file: "/home/oxidized/.config/oxidized/CE_Devices.db"
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      model: 1
      group: 2
      #username: 3
      #password: 4
    gpg: false

#-----Model map section, no need to re-configure------
model_map:
  comware: comware
#  Comware_qa: comware_qa
  comware_qa: Comware_qa
  Comware_qa: Comware_qa
  hp: comware
  #comware: comware_qa
  panos: panos
  fs: fs
  fs: fsos
  fsnsg: fsnsg
  fsos: fsos
  fsos: fs
  fs: cnos
  cnos: cnos
  cnos: fs

#----Hooks section, doesn't really need to be reconfigured at all other than adding/removing emails------#
hooks:

#This section is for emailing, config should not be touched other than adding or removing email addresses in the cmd: line. FOLLOW FORMAT.
  email_output:
    type: exec
    events: [post_store]
    change_pushed: '/opt/oxidized/oxidized-report-git-commits'
    cmd: 'bash /opt/oxidized/oxidized-report-git-commits'
#------------ADD EMAILS HERE WITH THE OTHERS (2x EACH ACCOUNT, ONE TIME AT THE BEGINNING AND ONE TIME AT THE END OF LINE)----------------------------------------
    cmd: "echo \"From: Oxidized\\nTo: <list of emails>\\nSubject: Config Change Detected for ${OX_NODE_NAME}\\nMIME-Version: 1.0\\nContent-Type: text/html; charset=utf-8\\n\\n\" > /tmp/${OX_NODE_NAME}.html; /opt/oxidized/oxidized-report-git-commits | tr -d '\r' | colordiff | ansi2html >> /tmp/${OX_NODE_NAME}.html; cat /tmp/${OX_NODE_NAME}.html | msmtp <list of emails>"
#----------------------------------------------------------------------------------------------------------------------
    async: true
    timeout: 5120
    path: /home/oxidized/.config/oxidized/config

#This section takes the config(s) grabbed and copies them to git and the "configs" file, doesn't need to be re-configured
  copy_configs:
    type: exec
    events: [post_store]
    cmd: 'cp -r /home/oxidized/.config/oxidized/DeviceConfigs.git/* /home/oxidized/configs/'
    cmd: 'git archive master --remote /home/oxidized/.config/oxidized/DeviceConfigs.git/ ${OX_NODE_NAME} | tar -x -C /home/oxidized/configs/'
    async: true
    timeout: 5120
    path: /home/oxidized/.config/oxidized/config

**------------End of Oxidized Full Config---------**

**--------------Example and handful of the affected configs saved locally----------**
-rwxr-xr-x  1 oxidized oxidized   12076 **Jul 29** 14:37 <device1>
-rwxr-xr-x  1 oxidized oxidized   41490 **Jul 29** 14:37 <device2>
-rwxr-xr-x  1 oxidized oxidized   12039 **Jul 29** 14:37 <device3>
-rw-r--r--  1 oxidized oxidized   38576 **Jul 29** 14:37 <device4>

[robertcheramy]

You have choosen the output "git".
The configs are saved in the git repo /home/oxidized/.config/oxidized/DeviceConfigs.git

Oxidized works only with one output.

If you want the file output, change the configuration:

output:
  default: file

[CootieCooter]

Okay that does make sense with the only one output, but we have a copy_config section (at the end of the supplied config above) that copies the files from the DeviceConfigs.git directory and pastes them to the saved local configs directory "/home/oxidized/configs/", which is where those saved copies have not been updated since July 29. I checked the DeviceConfigs.git directory and I don't see anything with an up-to-date recent modification date. Does oxidized expect a bare repository?

[robertcheramy]

Yes, oxidized expects a bare repository, which has been created by oxidized and not altered in any way. If you need to push the repository somewhere, use the github hook.
Fetching directly from the oxidized git bare repository could probably work, is not the supported way to to it.
If you have manipulated the oxidized git repository, oxidized may not work anymore.

[CootieCooter]

Okay, understood. We cloned the Deviceconfigs.git repository to be un-bare to help at least see the pulled device configurations. After troubleshooting more, I think our issue is that Oxidized is not committing updates to the DeviceConfigs.git repository. We have 2 other bare git repositories that were last updated with today's date or so (unfamiliar with those, must've been added by a co-worker), but this one, DeviceConfigs.git in the config file, that we have configured in the config file for Oxidized is still July 29, which is the same date for last update as the /home/oxidized/configs directory which is the main scope of issue. This theory is also supported by the copy_configs section thats in the config above, as we copy/paste from the DeviceConfigs.git to that configs directory, hence the correlation between the two sharing the same old date. Would you or others have any idea on how to go upon that???

[robertcheramy]

Try this: https://github.com/ytti/oxidized/blob/master/docs/Troubleshooting.md#oxidized-ignores-the-changes-i-made-to-its-git-repository

[CootieCooter]

I can't tell if Oxidized is even more broken now than what our initial issue was. It turns out that our DNS servers were updated on the Ubuntu 20.04.6 LTS server, as I was unable to ping our switches by their DNS names but was able to via their IPs, so I modified /etc/systemd/resolved.conf and included the new DNS server IPs and our domain, and then restarted it. I was then able to ping the switches by hostname, but oxidized was still failing to start. It was giving me this error:
<the ubuntu 20.04.6 server hostname>.<our domain> oxidized[2769961]: first 'failed to resolve path '/https:/github.com/<my github username>/comware.git': No such file or directory' was raised while opening git repo, then 'failed to make directory '/https:': Permission denied' was while trying to create git repo Dec 18 16:36:08 <ubuntu server>.<our domain> systemd[1]: oxidized.service: Main process exited, code=exited, status=1/FAILURE Dec 18 16:36:08 <ubuntu server>.<our domain> systemd[1]: oxidized.service: Failed with result 'exit-code'.

After tampering with it and doing ssh keys and such, i was able to finally get the oxidized user on the ubuntu server to do a git clone command from the remote repo, DeviceConfigurations, and have it cloned to /home/oxidized/.config/oxidized/test-repo. The repo path was also correct, as there is no "comware.git" anywhere in the oxidized config file. I was getting this error at first which made me generate a new SSH key and PAT token --> fatal: could not create work tree dir '/home/oxidized/.config/oxidized/test-repo': Permission denied.
I changed the repo path in the oxidized config file to use the PAT token I created on github developer settings under my profile's settings after this error told me that HTTP wasn't used anymore or something:
remote: Support for password authentication was removed on August 13, 2021. remote: Please see https://docs.github.com/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication. fatal: Authentication failed for 'https://github.com/<my github username>/DeviceConfigurations.git/'.

I then tried the sudo git clone command again and it prompted me to accept the ECDSA fingerprint, which I did say yes to. Then I restarted oxidized again after telling the output git repo path to use the remote DeviceConfigurations.git repo and it showed that oxidized started successfully and there were no errors when I did the "journalctl -fu oxidized" command. However, I checked our oxidizedLog.log and all the devices Oxidized was trying to connect to was throwing these kind of errors:
W, [2024-12-18T19:07:47.487201 #2779954] WARN -- : <switch ip> raised Net::SSH::HostKeyUnknown (rescued RuntimeError) with msg "fingerprint SHA256:<some key> is unknown for "<switch ip>"" W, [2024-12-18T19:07:48.438389 #2779954] WARN -- : comware/<hostname of switch> status no_connection, retry attempt 3 W, [2024-12-18T19:07:48.492587 #2779954] WARN -- : <switch ip> raised Net::SSH::HostKeyUnknown (rescued RuntimeError) with msg "fingerprint SHA256:<another key> is unknown for "<ip>"" W, [2024-12-18T19:07:49.439995 #2779954.

I am unsure how the existing ssh / host keys were removed or something, but I went ahead and SSH'd to several switches with the oxidized user account from the ubuntu / oxidized server and accepted the new keys. I then restarted oxidized again, checked the oxidizedLog.log file, and it is still throwing these errors for said switches that I just accepted the fingerprint to:
W, [2024-12-18T20:10:40.298263 #2783408] WARN -- : <ip of switch> raised Net::SSH::HostKeyUnknown (rescued RuntimeError) with msg "fingerprint SHA256:<another key> is unknown for "switch ip>"" W, [2024-12-18T20:10:41.215280 #2783408] WARN -- : comware/<hostname of switch> status no_connection, retry attempt 1 W, [2024-12-18T20:10:41.215448 #2783408] WARN -- : comware/<hostname of a different switch> status no_connection, retry attempt 1 W, [2024-12-18T20:10:41.266720 #2783408] WARN -- : 10.17.125.34 raised Net::SSH::HostKeyUnknown (rescued RuntimeError) with msg "fingerprint SHA256:<another different key> is unknown for "<different switch ip>"".

This is the new updated section of our oxidized config file now -->

output:
  default: git
  file:
    directory: "/home/oxidized/configs"
  git:
    #user: oxidized (Commented this out Dec 11, 2024 in regards to troubleshooting local stored config copies not updating)
    user: <My Name>
    #email: <old email> (Commented this out Dec 11, 2024 in regards to troubleshooting local stored config copies no>
    email: <my work email>
    #repo: "/home/oxidized/.config/oxidized/DeviceConfigs.git" (Commented this out Dec 11, 2024 in regards to troubleshootin>
    #repo: "https://github.com/<my github username>/DeviceConfigurations.git"
    #repo: "[email protected]:<my github username>/DeviceConfigurations.git"
    repo: "https://<my github username>:<my PAT key I created on github developer settings>@github.com/<my github username>/DeviceConfigurations.git"
    enabled: true
    auto_commit: true

If I didn't exactly say it before up above, I created a SHA256 SSH key on the Ubuntu 20.04.6 LTS server as the oxidized user, and added it to my github profile's SSH and GPG keys. At this point, our oxidized is even more broken than it was before. Because initially, it was still sending out emails of config differences and our other cloned git repos were being updated. But now, oxidized isn't polling any devices anymore and we haven't received any emails since yesterday (which is a problem because we have like 100 or so devices that we were getting emails for constantly all the time, so the days worth of quiet is indeed showing that there is an issue going on).

[CootieCooter]

I did make copies of 3 oxidized directories before I did all this troubleshooting, but I didn't backup anything related to ssh keys or .ssh/id_rsa or such, so if that is corrupted or the root issue, then I am unsure how to proceed.
Also, I run into this error often when trying to troubleshoot -->

Dec 18 23:07:17 <oxidized server>.<our domain>oxidized[2794214]: first 'failed to resolve path '/https:/<my PAT token>@github.com/<my github username>/comware.git': No such file or directory' was raised while opening git repo, then 'failed to make directory '/https:': Permission denied' was while trying to create git repo
Dec 18 23:07:17 <oxidized server>.<our domain> systemd[1]: oxidized.service: Main process exited, code=exited, status=1/FAILURE
Dec 18 23:07:17 <oxidized server>.<our domain> systemd[1]: oxidized.service: Failed with result 'exit-code'.

It's weird though since we don't have config.git anywhere in the config file, so confused why it keeps trying to use that path, and the permissions should be correct.

[robertcheramy]

You probably need single_repo; true in the config.
https://github.com/ytti/oxidized/blob/master/docs/Outputs.md#output-git