Output Git - how its working?

[Aanistratenko1984]

Hi community.

Please help me understand how the Output Git method works. I've read the manual, but couldn't find information about git push, commits, and other related operations.

It doesn't work automatically.

• In the config:

output:
  default: git
  debug: true
  git:
    user: oxidized
    email: [email protected] 
    repo: "/root/.config/oxidized/git-repos/default.git"

• We have 2 groups:

groups:
  dc:
    username: oxidized
    password: 
    vars:
      enable: true
  studios:
    username: oxidized
    password: 
    vars:
      enable: true

• Oxi automatically created local directories with files:

/var/docker/oxidized/git-repos # ls
dc.git  studios.git
/var/docker/oxidized/git-repos/dc.git # ls
config  description  HEAD  hooks  index  info  objects  refs  test.txt

• After reading configs from devices, Oxi updates files in the objects directory with names in hexadecimal:

/var/docker/oxidized/git-repos/dc.git/objects # ls
00  08  1f  2b  31  3e  43  46  53  5c  6f  7a  82  98  a4  aa  b1  b8  bf  c5  d6  d9  e5  ef  info
02  09  24  30  3a  3f  44  4d  5b  63  74  7d  96  99  a6  ac  b4  b9  c4  ce  d8  df  e9  fd  pack

• I did a manual Git init and added a remote:
  git remote add origin [email protected]:tech/oxidized-dc.git
• Checked Git status with default branch "main":

# git status
On branch main

• did manual push

 # git push origin main

• see all files in remote repository
  

Why doesn't Oxi automatically push files with the normal names of devices?
What branch Oxi use for automatically push configs ?
How does the rugged/libgit2 interface from the output section work?

[skiera6]

I'm not directly associated with Oxidized, but I can help clarify based on what I understand:

Why doesn't Oxi automatically push files with the normal names of devices?

Oxidized uses the rugged/libgit2 interface for interacting with Git repositories, which differs from the standard Git CLI behavior. It's important to note that normal Git hooks are not executed when using this interface. If you're expecting automatic pushes to a remote repository, you'll need to set up specific hooks within Oxidized's configuration.

Here’s an example configuration that uses a hook to push changes to a remote repository:

hooks:
  push_to_remote:
    type: githubrepo
    events:
      - post_store
    remote_repo: [email protected]:User/routers.git
    publickey: "~/.ssh/oxidized-key.pub"
    privatekey: "~/.ssh/oxidized-key"

In this setup, the post_store event triggers after a device configuration is stored, pushing the changes to the specified remote repository. Make sure to add your SSH keys (oxidized-key) to your remote Git repository to allow for authentication.

What branch does Oxi use for automatically pushing configs?

By default, Oxidized uses the master or main branch, depending on the repository setup. However, this can be configured in your Git settings if needed. You can manually specify the branch you want Oxidized to use by setting up your repository and remote accordingly.

How does the rugged/libgit2 interface from the output section work?

The rugged/libgit2 interface is a library-based approach to Git, used for handling commits, branches, and pushes in Oxidized. It works at a lower level compared to the standard Git command-line interface, which is why the behavior is a bit different. For example:

File changes are automatically committed locally in the .git/objects directory.
You must configure hooks within Oxidized to push changes to a remote repo, as the traditional Git hooks don't work with rugged/libgit2.

If you're seeing device configurations stored in the objects directory but not pushed automatically, this is due to the lack of Git hooks in the rugged/libgit2 setup, which needs to be configured manually as described above.

If anything i wrote is wrong feel free to correct me

[ts0818]

git is specified for output, but the bare repository that is created is the master branch.
Is it impossible to configure something like creating it in the main branch?
As far as "https://github.com/ytti/oxidized/blob/master/lib/oxidized/output/git.rb" is concerned,
It seems that you are using a library called rugged.
I would like to know if the following settings can be made on the Oxidized side.

config

output:
default: git
git:
user: Test
email: [email protected]
single_repo: true
repo: "/home/oxidized/.config/oxidized/demo"
# Is it not possible to specify a branch?
branch: main

[skiera6]

As far as i know it is not supported to do this in config. This might be helpful also if you are planing to use remote repo there's a wip feature here. Also i think you can edit your .git/config

[branch "master"]
    remote = origin
    merge = refs/heads/master

and change it to

[branch "main"]
    remote = origin
    merge = refs/heads/main

[ts0818]

@skiera6

Thank you for your cooperation.

In my project, the git push part is executed from a shell script, so
I want the branch to become main at the point of output (processed until commit).
Do you know how to deal with it?

I don't have any knowledge of Ruby, but for Oxidized processing, if you specify git in the output item in the config settings,
https://github.com/ytti/oxidized/blob/master/lib/oxidized/output/git.rb
I feel like a bare repository (brancg becomes master) is created.
I don't know if it can be overwritten with .git/config...

In other words,
https://github.com/ytti/oxidized/blob/master/lib/oxidized/output/git.rb
I don't know if it is possible to use the .git/config that I prepared for this process.

By the way, I use Oxidized with Docker Compose.

[skiera6]

Ok from my test this should work, test it and tell me did it work for you.

Please make sure to backup your repo

1. Save git repo in a volume mounted to your host.
2. From host go into the bare repo and run sudo git branch main
3. Than run sudo git symbolic-ref HEAD refs/heads/main
4. Than make sure your container can access HEAD and /refs/heads/main so run sudo chown user:group HEAD and sudo chown user:group /refs/heads/main as it is most likely owned by root after running git commands with sudo if you feel like it you can remove ref to previous branch in /refs/heads/

[ts0818]

@skiera6

Thank you for your cooperation.

The remote repository on GitHub is the main branch, so before running Docker Compose,
In the host directory mounted to the Docker container directory, just
git clone --bare
It worked fine.

Regarding the need to prepare a local Git repository in advance,
I hope it will be added to the documentation.

▼Below are the parts that may be related to this time.
■docker-compose.yml

---
version: "3"
services:
  Oxidized:
    restart: always
    image: app-oxidized:v1.0
    ports:
      -8888:8888/tcp
    environment:
      CONFIG_RELOAD_INTERVAL: 600
    volumes:
       - type: bind
         source: /opt/app/oxidized/.config/oxidized
         target: /home/oxidized/.config/oxidized

■/opt/app/oxidized/.config/oxidized/config

...omission

output:
  default: git
  git:
    user: Oxidized
    email: [email protected]
    repo: "~/.config/oxidized/test-personal-github-app.git"

■Prepare a bare repository in advance

cd /opt/app/oxidized/.config/oxidized
git clone --bare https://<your account>:<PAT (Personal Access Token)>@github.com/<your account or organization>/test-personal-github-app.git

■Start Oxidized with Docker Compose

cd /home/[user]/work/app
ls
  docker-compose.yml
  image/oxidized

docker-compose up -d oxidized

[ts0818]

@skiera6

After that, there seems to be a problem where the contents of the remote repository are overwritten during Oxidized processing.
For example, a repository that used to manage 10 files on GitHub ends up being reduced to 1 file.
(Perhaps this is due to changing the number of target devices in router.db?)

Do you know how to avoid this?

[skiera6]

@ts0818

Regarding the need to prepare a local Git repository in advance,
I hope it will be added to the documentation.

There's no need to create local git repo in advance further more it's discouraged.

• Non-Bare Repository: Contains a working directory with the checked-out files, allowing you to edit, add, and commit changes directly.
• Bare Repository: Does not have a working directory. It consists solely of the Git version control information.

if you plan on using remote repo with oxidized use

hooks:
  push_to_remote:
    type: githubrepo
    events:
      - post_store
    remote_repo: [email protected]:User/routers.git
    publickey: "~/.ssh/oxidized-key.pub"
    privatekey: "~/.ssh/oxidized-key"

as you basically try to implement an existing feature with your script.

For example, a repository that used to manage 10 files on GitHub ends up being reduced to 1 file.
(Perhaps this is due to changing the number of target devices in router.db?)

• You clone only the .git data from your repo
• Oxidized might be saving configurations from all input sources, but without properly merging or tracking existing files, it could overwrite the repository state, leaving only the latest file.

if you want to use standard git repo as your backup tool and do it with a script just use output: file and save the files in a repo that you cloned or let the oxidized handle it with the hook

[ts0818]

@skiera6

Thank you for your cooperation.

It is necessary to use "proxy" to "git push", and unfortunately "git push" using "hooks" does not support "proxy".
Not available.

The focus is on setting output:git in Oxidized's config,
I don't know if it's an Oxidized problem or a rugged problem,
This means that a bare repository will be created in the master branch.

We understand that there is no clear solution to this issue.

[robertcheramy]

Rugged does support the use of a :proxy_url in fetch and push. Oxidized doesn't. If this feature is needed, someone need to implement it.

I've read the thread. Do NOT fiddle with oxidized internal repository. Do not create it, do not modify it. You will break things. The only acceptable changes are the one documented (git gc and rename the git branch) and only while oxidized is not running.
If you need to push the repo somewhere, use the githubrepo hook. If you need to push through a proxy, use the githubrepo hook to push to a local repo, which you can then push though a proxy. Or make a separate issue requiring the feature that the githubrepo hook supports proxies.

[ts0818]

@robertcheramy

Thank you for your cooperation.

You updated the document on October 15, 2024.

34c3e7d

https://github.com/ytti/oxidized/blob/master/docs/Hooks.md

Githubrepo will use the branch name used in the git output as a remote branch name.
When creating the git repository for the first time, Oxidized uses the default branch name configured in git with git config --global init.defaultBranch . The default is master.

In Oxidized's Dockerfile,
RUN git config --global init.defaultBranch
Does this mean that I can specify the branch if I do something like this?

For example, if you want it to work on the main branch, use Oxidized's Dockerfile.
RUN git config --global init.defaultBranch main
I'll keep it.

If you need to push the repo somewhere, use the githubrepo hook.
If you need to push through a proxy, use the githubrepo hook to push to a local repo, which you can then push though a proxy.
Or make a separate issue requiring the feature that the githubrepo hook supports proxies.

I would appreciate it if you could tell me the specific settings of the config file to achieve this.
In my case, I need to proxy from a physical machine to a VM in Azure to git push to GitHub.

[robertcheramy]

Setting init.defaultBranch only works if your ~/.gitconfig ist persistend, which I doubt in the docker container. You'll probably have to rename de branch manually as described at https://github.com/ytti/oxidized/blob/master/docs/Hooks.md#custom-branch-name. You should also use the latest oxidized docker image, as 0.30.1 did not fully support custom branch names.

Regarding the proxy thing. You are doing something very specific which is not supported by oxidized. You have already become a lot of help from @skiera6 (Thanks a lot by the way @skiera6) within this issue . You've got everything you need to solve your problem. Read the docs, reread the issue, try it out and find a solution for your problem yourself.

[ts0818]

@robertcheramy

Thank you for your cooperation.

In my case, Oxidized is being used in a way that is not expected.
I understand.