Skip to content

Latest commit

 

History

History
223 lines (168 loc) · 8.44 KB

README.md

File metadata and controls

223 lines (168 loc) · 8.44 KB

fluent bit HSDP logging output plugin

This plugin outputs your logs to the HSDP Host Logging service. This is useful when your workloads are not running on Cloud foundry, but you still want to utilize the central logging facilities of HSDP.

Fluent bit supports parser and filter plugin which can convert unstructured data gathered from the log Input interface into a structured one and to alter existing structured data before ingestion.

More on fluent-bit

Cloud and On-Premise

The plugin supports deployment to both Cloud and On-Premise environments. Depending on the deployment type you can either specify the Cloud Region and Environment or the On-Premise IamUrl and IdmUrl values.

API Signing, Service Identities or Logdrainer

The plugin supports both the API Signing authorization mechanism or the use of a IAM Service Identity with the LOG.CREATE scope. It also supports using a CF Logdrain endpoint although that limits the log format output (e.g. no support for the custom field)

Configuration options

Your fluent-bit.conf file should include an entry like below to enable the plugin:

[output]
    Name hsdp
    Match *

Configuring the authorization mechanism and HSDP Logging endpoints should ideally be done by setting the right Environment variables:

Settings

General

These keys are relevant when using either SigningKey or Service identities

Key Description Environment variable Required
ProductKey The Product key of your proposition HSDP_PRODUCT_KEY Optional
IngestorHost The HSDP ingestor host HSDP_INGESTOR_HOST Optional
Debug Shows request details when set to true HSDP_DEBUG Optional
CustomField Adds the field hash to custom field when set to true HSDP_CUSTOM_FIELD Optional
InsecureSkipVerify Skip checking HSDP ingestor TLS cert. Insecure! HSDP_INSECURE_SKIP_VERIFY Optional

Signing keys

Key Description Environment variable Required
SharedKey The Shared key for signing requests HSDP_SHARED_KEY Optional
SecretKey The Secret key for signing requests HSDP_SECRET_KEY Optional

Service identities

Key Description Environment variable Required
Region The HSP Region (Cloud) HSDP_REGION Optional
Environment THE HSP Environment (Cloud) HSDP_ENVIRONMENT Optional
IamUrl The IAM URL (On-Premise) HSDP_IAM_URL Optional
IdmUrl The IDM URL (On-Premise) HSDP_IDM_URL Optional
ServiceId The Service ID to use for authentication HSDP_SERVICE_ID Optional
ServicePrivateKey The Service private key HSDP_SERVICE_PRIVATE_KEY Optional

Logdrain

You can reuse an existing Cloud foundry logdrainer endpoint to ship your logs to HSDP logging. The advantage is that you only need the Logdrain URL itself, no other configuration. This URL is considered a credential so care should be taken to protect it though.

Key Description Environment variable Required
LogdrainUrl The HSP Logdrain URL HSDP_LOGDRAIN_URL Optional

Environment variable values take precedence over those in configuration files.

Record field mapping to HSDP logging resource

The plugin supports full pass-through of the native LogEvent JSON message type, example:

{
  "resourceType": "LogEvent",
  "id": "7f4c85a8-e472-479f-b772-2916353d02a4",
  "applicationName": "OPS",
  "eventId": "110114",
  "category": "TRACELOG",
  "component": "TEST",
  "transactionId": "2abd7355-cbdd-43e1-b32a-43ec19cd98f0",
  "serviceName": "OPS",
  "applicationInstance": "INST‐00002",
  "applicationVersion": "1.0.0",
  "originatingUser": "SomeUsr",
  "serverName": "ops-dev.cloud.pcftest.com",
  "logTime": "2017-01-31T08:00:00Z",
  "severity": "INFO",
  "logData": {
    "message": "VGVzdCBNZXNzYWdl"
  },
  "custom": {
    "foo": "bar"
  }
}

NOTE: the logData.message field must be base64 encoded

field mapping

Alternatively, the plugin maps certain record fields to defined HSDP logging resource fields. The below table shows the mapping, and the default value.

Record field HSDP logging field Default value Details
server_name serverName fluent-bit
app_name applicationName fluent-bit
app_instance applicationInstance fluent-bit
app_version applicationVersion 1.0
category category TraceLog
severity severity informational
service_name service_name fluent-bit
originating_user originating_user fluent-bit
event_id event_id 1
transaction_id transaction_id random UUID if original input is not a valid UUID a new one will be generated
trace_id trace_id
span_id span_id
logdata_message logData.Message field hash will replace the default field hash dump went present

Fields mapped to a HSDP logging resource field will be removed from the log message dump

The below filter definition shows an example of assigning fields

[filter]
    Name record_modifier
    Match *
    Record server_name ${HOSTNAME}
    Record service_name Awesome_Tool
[filter]
    Name modify
    Match *
    Copy container_name app_name
    Copy container_name service_name
    Copy component_name component
    Copy container_id app_instance

Remaining fields will be rendered to a JSON hash and assigned to logData.Message

Building

docker build -t fluent-bit-out-hsdp .

Testing with Docker

docker run --rm \
    -p 127.0.0.1:24224:24224 \
    -e HSDP_PRODUCT_KEY=product-key-here \
    -e HSDP_REGION=us-east \
    -e HSDP_ENVIRONMENT=client-test \
    -e [email protected] \
    -e HSDP_SERVICE_PRIVATE_KEY="$(cat service_private_key.pem)" \
    -it philipssoftware/fluent-bit-out-hsdp:latest

Once the above is running you can start other Docker containers and use fluentd log driver to start logging to HSDP logging:

docker run --rm -it --log-driver fluentd alpine echo "hello world"

Helm Chart

You can deploy fluent-bit and the HSDP plugin using a Helm chart

Secret

The chart will attempt to read credentials from an hsdp-logging Kubernetes secret which should reside in the namespace. An example hsdp-logging-secret.yaml is included below. Make sure you replace the values accordingly

apiVersion: v1
kind: Secret
metadata:
  name: hsdp-logging
type: Opaque
data:
  shared_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz
  secret_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz
  product_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz
  ingestor_host: aHR0cHM6Ly9sb2dpbmdlc3RvcjItY2xpZW50LXRlc3QuZXUtd2VzdC5waGlsaXBzLWhlYWx0aHN1aXRlLmNvbQ==

Apply the secret to the right namepace:

kubectl apply -f hsdp-logging-secret.yaml -n logging

Finally, install the Helm chart:

helm repo add loafoe https://loafoe.github.io/helm-charts/
helm install fluent-bit-out-hsdp loafoe/fluent-bit-out-hsdp -n logging

If the credentials are correct you should now see your Kubernetes cluster logs in the HSDP Logging system.

Contact / Getting help

Andy Lo-A-Foe [email protected]

License

License is MIT