Agnostic Vars

Table of Contents

• Description
  • Features
  • Example of agnosticv repositories
• agnosticv CLI
  • Install
  • Usage
• Build
• File naming convention
  • Nomenclature
  • Common files
  • Includes
  • Meta files
  • Related Files
  • Leaf files
• Merging strategies
  • Default
  • Custom merge strategies
• See also

Description

Agnosticv is a generic way of organizing and merging YAML files to manage a Catalog.

Currently the project is composed of:

1. CLI agnosticv, a generic implementation.

2. agnosticv-operator an OpenShift operator that uses the CLI and generates OpenShift Custom Resources for Babylon based on the content of the agnosticv repositories. The operator is more opinionated than the CLI as it expects a specific file structure in the agnosticV repository. We use it to manage the catalog of our demo portal RHPDS.

Features

• Merge YAML files automatically following a particular convention, see below.

• Support for includes.

• Inject information about last change (commit, author, date) when merging the vars of a catalog item

• When listing catalog items, filter catalog items using JMESPath expressions (--has flag)

• Configure custom policies or behavior, per repository:

  • Configure merge strategies

  • Validate variables using OpenAPI v3 schemas. Just place schemas in a .schemas directory at the top-level of the agnosticv repository.

  • Configuration file /.agnosticv.yaml

Example of agnosticv repositories

1. public: see in cli/fixtures for an example of structure of a catalog

2. private GPTE agnosticv repository

3. private Red Hat Summit 2021 repository

4. private 2020Summit-AgnosticV repository

5. private RHTR_2020_agnosticV repository

agnosticv CLI

Install

Download the binary for your architecture (linux, mac, windows, ..) from release page.

Usage

The CLI agnosticv has the following capabilities:

• list all the catalog items present in a directory

• merge and print the vars of an item of the catalog

Usage

Usage of agnosticv:
  -debug
    	Debug mode
  -git
    	Perform git operations to gather and inject information into the merged vars like 'last_update'. Git operations are slow so this option is automatically disabled for listing. (default true)
  -has value
    	Use with --list only. Filter catalog items using a JMESPath expression.
    	Can be used several times (act like AND).

    	Examples:
    	--has __meta__.catalog
    	--has "env_type == 'ocp-clientvm'"
    	--has "to_string(worker_instance_count) == '2'"

  -list
    	List all the catalog items present in current directory.
  -merge string
    	Merge and print variables of a catalog item.
  -or-related value
    	Use with --list only. Same as --related except it appends the related files to the list instead of reducing it.

    	Example:
    	--list --related dir/common.yaml --or-related includes/foo.yaml
    	   List all catalog items under dir/ and also all catalog items that include includes/foo.yaml

    	Can be used several times (act like OR).
  -related value
    	Use with --list only. Filter output and display only related catalog items.
    	A catalog item is related to FILE if:
    	- it includes FILE as a common file
    	- it includes FILE via #include
    	- FILE is description.adoc or description.html

    	Example:
    	--list --related dir/common.yaml --related includes/foo.yaml
    	   List all catalog items under dir/ that also include includes/foo.yaml

    	Can be used several times (act like AND).
  -root string
    	The top directory of the agnosticv files. Files outside of this directory will not be merged.
    	By default, it's empty, and the scope of the git repository is used, so you should not
    	need this parameter unless your files are not in a git repository, or if you want to use a subdir. Use -root flag with -merge.
  -validate
    	Validate variables against schemas present in .schemas directory. (default true)
  -version
    	Print build version.

List catalog items in local directory

cli $ ./agnosticv --list
fixtures/gpte/OCP_CLIENTVM/dev.yaml
fixtures/gpte/OCP_CLIENTVM/prod.yaml
fixtures/test/BABYLON_EMPTY_CONFIG/dev.yaml
fixtures/test/BABYLON_EMPTY_CONFIG/prod.yaml
fixtures/test/BABYLON_EMPTY_CONFIG/test.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_AWS/dev.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_AWS/test.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_OSP/dev.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_OSP/prod.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_OSP/test.yaml

Merge and print the vars of a catalog item

cli $ ./agnosticv --merge fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml
---
# MERGED: (1)
# fixtures/common.yaml
# fixtures/test/account.yaml
# fixtures/test/account.meta.yaml
# fixtures/test/BABYLON_EMPTY_CONFIG_AWS/common.yaml
# fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml
__meta__:
  deployer:
    scm_ref: test-empty-config-test-0.5
    scm_type: git
    scm_url: https://github.com/redhat-cop/agnosticd.git
    type: agnosticd
  secrets:
  - from-top-common.yml
  - name: gpte
  [...] output omitted

1. Merge list: gives information about how files were merged to produce the final set of variables

Note	common.yaml files are always included when merging. agnosticv searches for those files as long as it is in the same git repository. If the files are not versioned with git, it is possible to "chroot" the search using the --root parameter.

Build

cd cli
go get
go build -o agnosticv

File naming convention

Nomenclature

• Catalog item, or Leaf: a file considered a catalog item. It appears when listing an agnosticv repository with agnosticv --list. To print the content of a catalog item, you run agnosticv --merge PATH. That merges common files, meta files, included files, and finally, the leaf to produce the catalog item.

• Common file: a file that is automatically included in the merge list. Ex: common.yaml

• Related file: a file that is related to a catalog item. Ex: description.adoc

• Included file: any file that is included in the merge list using the #include PATH feature

• Meta file: Any file ending with .meta.yml or .meta.yaml. It contains the value of the meta dictionary.

Common files

Some files are automatically included in the merge list to produce the final catalog item. The following names are valid common YAML files:

• common.yaml

• common.yml

• account.yaml

• account.yml

They can be placed at any level in the agnosticv repository.

Includes

Files included in the merge list using the #include PATH feature. See the dedicated section below.

Usually, we place them in an includes/ directory at the top of the agnosticv repository.

#include merge feature

• syntax: #include FILENAME

• where: In any file

• identation is ignored

  #include /file.yaml

  And:

      #include /file.yaml

  are the same.

• FILENAME is added to the merge list right before current file regardless of the position of #include in the file. In other words, current file vars take precedence over included files vars.

  • That’s also why you should put all your includes at the top of the file.

• if FILENAME starts with / then path is absolute to the AgnosticV repo.

  • if not, the path is relative to the current file

Example

gpte/OCP4_WORKSHOP/prod.yaml

#include /includes/file1.yaml
#include /includes/file2.yaml

cloud_provider: ec2
key_name: opentlc_admin_backdoor
repo_method: file

subdomain_base_suffix: .example.opentlc.com
HostedZoneId: Z3IHLWJZOU9SRT

agnosticv_meta:
  deploy_with: babylon

includes/file1.yaml with vars at the "agnosticd" level

var1: value1
var2: value2

includes/file2.yaml with meta vars

agnosticv_meta:
  secrets:
    - somesecret

__meta__:
  secrets:
    - name: somesecret
      namespace: gpte

The merge list will be:

1. /common.yaml

2. /gpte/account.yaml

3. /gpte/OCP4_WORKSHOP/common.yaml

4. /includes/file1.yaml

5. /includes/file2.yaml

6. /gpte/OCP4_WORKSHOP/prod.yaml

Meta files

For any common file, leaf file, or included file, you can create an associated meta file to be automatically included. The meta file can contain the value of the __meta__ dictionary. It is convenient to separate that special dictionary from the other variables.

For example, the following files are valid meta files:

• common.meta.yml meta file for common.yml

• account.meta.yml meta file for account.yml

• dev.meta.yml meta file for dev.yml

Warning

you can only put the content of the __meta__ variable in a meta file.

example1: content of meta file

__meta__:
  secrets:
    - name: mysecret

example2: with the content of __meta__ directly, same as example1

secrets:
  - name: mysecret

example3: wrong meta file - This will fail

__meta__:
  secrets:
    - name: mysecret

another_var: value (1)

1. other variables than __meta__ are not allowed

Related Files

• Related files are helpful in estimating when a catalog item was last changed. All catalog items related to that file are considered touched if a related file is touched.

• To print all catalog items related to a file, run agnosticv --list --related-to RELATED_FILE

• All included files of a catalog item are automatically considered related files.

• All common files of a catalog item are automatically considered related files.

• It is possible to add custom related files using the config file; see section below.

• It is possible to load related files into the merged vars using the config file; see section below.

Configuration

You can add custom related files using the related_files configuration option in a .agnosticv.yaml configuration file. It contains a list of filenames that if present in the same directory of a catalog item, will be considered as related to that catalog item.

You can decide to load the content of the relative file into a path by specifying the path as a JSON Pointer format. The destination path will be a dictionary and the content will be loaded into the 'content_key'.

Here is an example of configuration:

/.agnosticv.yaml

# For any catalog item, consider those files in same directory as related files:
related_files_v2:
  - file: service-ready-message-template.html.j2
    load_into: /__meta__/catalog/message_templates/service_ready
    content_key: template
    set:
      format: jinja2
      output_format: html
  - file: description.txt

When merging, that will produce:

__meta__:
  catalog:
    message_templates:
      service_ready:
        format: jinja2
        output_format: html
        template: |
          <<content of the file>>

Leaf files

The "leaf" files, or catalog items, are just the rest of the YAML files, having one of these extensions:

• yml

• yaml

You can list all catalog items in a directory by using --list parameter: agnosticv --list

Files ignored

• Any dotfile is ignored. Ex: .git

• Any directory named includes is reserved to includes. The files in those directories are never considered as catalog items.

• Any file containing:

  #agnosticv catalog_item false

  is ignored. It is not considered a catalog item.

Merging strategies

Default

When it comes to merging variables, there are different possible strategies.

The default is the following:

What	Dictionaries	Lists	Strings / Numbers
__meta__ and agnosticv_meta dictionaries	merge	append	replace
Rest of the vars Same behavior as if you were using ansible extra vars	replace	replace	replace

Custom merge strategies

It is possible to extend agnosticV and define the merge strategy to use on what variable or part of a dictionary variable.

To do that, you can define the custom merge strategies in any schema in the .schemas directory. Just use the x-merge keyword at the beginning of the schema. x-merge is a list of strategies. Each strategy defines a path and a strategy name to apply to that path. path is a JSON Pointer. For the list of strategies, see section below.

.schema/schema.yaml example of x-merge custom strategy

type: object
x-merge:
  - path: /__meta__/access_control # (1)
    strategy: overwrite # (2)
properties:

1. The path of the variable or key of dictionnary, as a JSON Pointer, to apply the custom strategy against.

2. When merging, agnosticv will overwrite the content of __meta__.access_control instead of merging it.

For example, with the schema above and following merge list:

# MERGED:
# fixtures/common.yaml
# fixtures/test/account.yaml
# fixtures/test/BABYLON_EMPTY_CONFIG/common.yaml
# fixtures/test/BABYLON_EMPTY_CONFIG/prod.yaml

The value of __meta__.access_control from prod.yaml will take precedence and overwrite.

Here are the available custom merge strategies:

Strategy	Can be applied to	Dictionaries	Lists	Strings / Numbers
overwrite	List or Dict	replace	replace	replace
merge	List or Dict	Merge	Append	replace
merge-no-append	Dict	Merge	replace	replace
strategic-merge	List or Dict	Strategic Merge [1]	Strategic Merge [1]	replace

See also

• AgnosticD deployer

• Babylon project

---

1. Merge similar to kubernetes stategic merge patch. The patch merge-key for list is name.