An idea of the devops infrastructure at Tenzir

[GTrunSec]

Due to my current time constraints, I can only complete some concepts for this work. Therefore, I presented some previously accomplished work. Furthermore, I will continue to refine and test this framework during sporadic moments of idle time to ensure its flawless execution in the intended tasks. Learning MLOps has consumed most of my time. Maybe after two months, I can continue this work.

[mavam]

Closing this for now. Feel free to reopen if still relevant.

[GTrunSec]

After a period of experimentation and verification, I have explored a systematic approach to managing a complex DevOps framework environment effectively. We can provide users and developers with a packaged and configurable DevShell environment, which can be accessed through various methods such as Docker, DevContainer, and Nix. Additionally, the user's interface can be managed and deployed with reproducible configurations through GitOps in an individual repository.
Within the apiece shell, we can encapsulate documentation within relevant commands, allowing users to easily access the associated documentation through the command's flags. In the framework, we extensively utilize a file system-based structure management and submodule presets, aimed at simplifying the management of developer-oriented profiles and user interfaces.

• https://github.com/numtide/prj-spec Directory Specification, but for the project
• WIP

[GTrunSec]

• tracking the DevOps branch: https://github.com/GTrunSec/vast/tree/devops/nix/devops

experimental paradigm in DevOps

you can run as follows command to try that.

• docker/podman

docker pull ghcr.io/gtrunsec/tenzir-devshell-dev:latest
git clone https://github.com/gtrunsec/vast -b devops
cd vast
docker run --platform linux/amd64 -it -v "$(pwd):/work" --entrypoint bash ghcr.io/gtrunsec/tenzir-devshell-dev:latest
podman run --platform linux/amd64 -it -v "$(pwd):/work" --userns=keep-id --entrypoint /bin/bash --entrypoint bash ghcr.io/gtrunsec/tenzir-devshell-dev:latest

• nix

git clone https://github.com/gtrunsec/vast -b devops 
cd vast && direnv allow

[GTrunSec]

for developers

• Components:
  • process-compose-sqlit-example (devshell + std + flakeParts + nixosModues(for declaring the service, similar with devenv, but we use the flakeParts + devshell))

• jupyenv-example (jupyterlab + nixosModules + jupyenv)
  • jupyenv-example (lab|--help)
    

For users

• WIP

[GTrunSec]

Std TUI / CLI:

• exploring all of jobs/task/actions with std-tui. Just run std in the devshell to get a look.

Also, we can extend the act of actions for one command. How about having more acts for the oci-image/tenzir-static
std //repo/oci-images/tenzir-static:build -> std //repo/oci-images/tenzir-static:test(for testing/debuging)

[GTrunSec]

DONE the first minor version for the interface of users. -> https://github.com/GTrunSec/tenzir-users-devops-template

[mavam]

This looks cool! Can you help us describe what use cases you are going after with this effort?

[GTrunSec]

today ,I have successfully developed friendly DevOps frameworks for users and developers. For a framework, it should cover a wide range of user scenarios and use cases, whether simple or complex. For instance, providing a software supply chain interface similar to "makes," offering a reproducible Jupyter environment like "jupyenv," enabling a quick launch of a series of services (like Docker Compose) through using "process-compose," or delivering a devshell scenario to showcase new features/latest changes of the software swiftly. Besides, my current design mainly provides a framework and theory that allows users to store all states of the products and ecosystems(toolchains) based on GitOps.
From a user's perspective, simplicity, understanding, and well-considered design are essential foundations. Therefore, I have simplified the use cases formed in the user-facing interfaces as much as possible to ensure that the APIs are straightforward and clear.

[GTrunSec]

I will take the time to update and provide additional information on practical use cases and API usage in the upcoming period. As for the complex concepts surrounding framework theory and design, I have included some relevant information in the "users-template" for your reference.

[GTrunSec]

Developers-interface: 0.1 version finished today -> https://github.com/GTrunSec/vast/tree/8975b8516c5c69ba96b9f71a206d7b0896057930/nix/devops

[GTrunSec]

For the users:

Folders structures

• Q: What's the minimal devshell? How many files will be contained in that path?
  A: 1. you may find the minimal template here: https://github.com/GTrunSec/tenzir-users-devops-template/tree/template
  2. only four files will be contained by default, which are gitignore, flake.nix, flake.lock, devshell.toml

• Q: If I created a new case, which folder should I put in? What are the folders for "dev" and "prod"? Why are they named that way?
  A: The folders "dev" and "prod" can serve as a means of categorization or scenario distinction. For instance, when a user creates a script, they should assess whether it will be executed in a production or development environment. Overall, this allows for a more structured organization of the repository.

std //prod/tasks/deploy-tenzir-aws:run -> //<prod/dev>/<tasks>/<name of task>:<action>

• Q: How do I update or keep the template's status constant?
  A: when you enter a deshell, you can use the nix flake lock --update-input tenzir-devops, and the flake.lock will keep the status constant.

Tasks

Don't forget git add after your changes. Remember, we are under GitOps.

Q: How do I create a script/task to run my aws deployment? Is it complex?
A: not complex, just copy this example to a new folder, then replace the entrypoint and env with your own, that's all

• output

std prod/tasks/deploy-tenzir-aws:run
-------------------------------------------------------------------------------------------------------
Executing /home/guangtao/ghq/github.com/GTrunSec/tenzir-users-devops-templates/.local/state/last-action
With args []
-------------------------------------------------------------------------------------------------------
[INFO] Making environment variables for makeEnvVarsOutput:
Note: AWS CLI version 2, the latest major version of the AWS CLI, is now stable and recommended for general use. For more
information, see the AWS CLI version 2 installation instructions at: https:docs.aws.amazon.com/cli/latest/userguide/inst
all-cliv2.html

usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help

Q: Can I use json/toml/yaml to write a task/job?
A: Yes, you can. But the nix will give it full feature support, JSON,TOML,yaml are just gives it the simplest declaring for running. example -> https:github.com/GTrunSec/tenzir-users-devops-template/tree/e4110a96cf97d6c3030633e34d319cd0bbfdf8e7/dev/tasks/script-json

WIP.

[GTrunSec]

The new documentation framework and version will be released by the end of next month, driven by emanote.

It will provide a user-friendly command-based documentation system, facilitating users to query and write tasks, as well as providing guidelines for creating their own tasks.

https://hardenedvault.net/blog/2023-08-25-hardened-siem-xdr-ved/

Hardenedvault is expected to be the first user of the DevOps framework and is anticipated to offer a community preview version of Tenzir & Vault early next year. The framework will include guidelines on maintaining a highly structured repo, such as IDS's rules, query languages, and feed lists, and deploying and testing various targets/envs.

I mainly use d2 as the diagramming language, and each chain job will attach a d2 file to describe that visual workflow. Furthermore, use cases integrating Prefect with Tenzir and DataOps and an AWS user instance version will be provided. Considering the complexity, the transition from Tenzir-k8s will be considered using https://kustomize.io/ to https://kubevela.io/ as an initial step.