A utility to generate a Splunk knowledge & alerting app from YAML declarations
generate.py will generate an application and takes several arguments, found with the --help metaargument
YAML files and .csv static lookups are placed in a source directory ( default /src ) with a format described below and processed in to relevant Splunk .conf stanzas.
placing a savedsearches.conf file with a default stanza in a directory allows extra fields to be applied to saved searches (alert or report) to facilitate custom actions. This template is applied to any searches within that directory and any subdirectories unless overridden by another savedsearches.conf. These are not applied to the global savedsearches.conf default stanza (as that might cause them to apply to other searches in the app when they are not intended) but are instead added to each individual item
Files that are to be included in the application such as UI elements, app icons, etc are placed in the app skeleton directory ( default skel/ ) and are copied first in to the resulting app, and then appended to by the generator
A Makefile and Jenkinsfile are included for convenience
make accepts the following environment variable arguments:
- APPNAME (defaults to GeneratedApp)
- APPVERSION (defaults to 1.0.0 )
- SRCDIR (defaults to src)
- APPSKEL (defaults to skel)
- LOOKUPDIR (defaults to src/lookups)
Jenkinsfile is configured to use the secrets splunkbase and splunk_acs, for login credentials to splunkbase/splunk.com to run appinspect, and ACS for deployment to a Splunk Cloud stack and the environment variable APPNAME for the name as it appears in Splunk
Fields are defined in Splunk per sourcetype ( see Splunk's documentation for props.conf ) under the fields key
fields is optional in a sourcetype document, and contains a key-value mapping with values of string, which are converted to eval field calculations.
A special case of field operations is aliases, as an optional top level key pointing to a single level key-value mapping that are converted to a single alias definition
sourcetype documents may contain a tags key, pointing to an array of tags to be applied by Splunk to the sourcetype
sourcetype documents may contain a lookups key, for automatically applying lookups to a sourcetype. Lookups may contain inputs and outputs, either as a list (passed to lookup unmodified) or a single level key-value mapping (passed to lookup as value AS key. SPL is somewhat counterintuitive in the order of the AS operation)
sourcetype: "new_sourcetype"
aliases:
foo: bar
fields:
myfield: coalesce(first, second)
lookups:
thing_to_other_thing:
inputs:
- infield
outputs:
outfield: realfield
tags:
- mytag
This will output a props.conf entry:
[new_sourcetype]
EVAL-myfield = coalesce(first, second)
FIELDALIAS-Global = bar ASNEW foo
LOOKUP-thing_to_other_thing = thing_to_other_thing infield OUTPUT realfield AS outfield
and a tags.conf entry:
[sourcetype=new_sourcetype]
mytag = enabled
CSV files (files with a .csv extension) present in the sources directories will be interpreted as static lookup files and applied to transforms.conf without the .csv extension
eventtype documents must include search and may optionally include tags, pointing to an array of tags to be applied by Splunk to the eventtype
a macro document contains the macro field as the name and a definition field as the SPL definition of the macro
Search definitions Must at minimum include a name and search field.
A savedsearches.conf file with a [default] clause in a directory is used as a template for savedsearches.conf clauses for every search in that directory and any child directories. i.e., every search clause generated will contain the fields in the [default] entry, unless overridden by the yaml file itself.
SplunkGen differentiates between a report and an alert if after applying default configuration ( a local or parent directory savedsearches.conf ) and parsing the file if action counttype or any risk annotation keys are present it is treated as an alert, unless it is explicitly declared not to be so with alert: false
Search scheduling is declared with top level cron, earliest and latest fields, which default to the values in DEFAULT_SEARCH_SCHEDULING in defaults.py
Reports are searches without alert actions, either in the template or definition. A report will send it's results to a lookup. If the search does not contain an outputlookup command SplunkGen will append one, using the search name as the lookup name as a CSV file.
Reports will also add a transforms.conf entry for the output, removing the trailing .csv file extension. This behaviour can be suppressed by adding a no_transform: true field to the document (if for instance the report outputs to a key-value store in another app)
The format is based on Splunk's ESCU generator tool and files generated by that tool remain compatible.
Alerts may contain a top level trigger object containing a relation value as documented by Splunk savedsearches.conf as a key to an integer value. Defaults to greater than: 0
The tags object may contain information relating to risk & Splunk ES correlation searches & notables. Available keys are documented in the ANNOTATION_KEYS constant in util.py
Fields available in the tags object may be provided at the top-level object
Splunk allows alerts to be throttled on a single field and this is exposed as a throttle object containing fields as a list or string scalar to throttle the alert on, and a suppression period, time
Risk-based alerting in ES can be achieved through risk_score, impact, and confidence fields, all of which are optional and can be placed in the tags object or at the top level document. Defaults to 100, and risk_score is calculated as impact * confidence / 100 if not explicitly declared.
An observables object is used by Splunk to apply risk to specific objects by way of a risk alert action and if an observables object is present, SplunkGen will add RBA fields to the final representation
Notables are used in ES as an alert action, and unless suppressed with notable: false SplunkGen will generate notable action fields for alerts, with severity the same as the RBA risk score