Skip to content

rchicoli/docker-log-elasticsearch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Docker Log Elasticsearch

GoDoc License: Apache 2.0 Release Travis Status Coveralls Coverage Go Report Card

docker-log-elasticsearch forwards container logs to Elasticsearch service. The current application release is alpha. (see Roadmap).

Releases

Branch Name Docker Tag Elasticsearch Version Remark
master 1.0.x 1.x, 2.x, 5.x, 6.x Future stable release.
development 0.0.1, 0.2.1 1.x, 2.x, 5.x, 6.x Actively alpha release.

Getting Started

You need to install Docker Engine >= 1.12 and Elasticsearch application. Additional information about Docker plugins can be found here.

How to install

The following command will download and enable the plugin.

docker plugin install rchicoli/docker-log-elasticsearch:latest --alias elasticsearch

How to configure

The plugin must be disabled in order to change the plugin settings

Environment Description Default Value
LOG_LEVEL log level to output for plugin logs (debug, info, warn, error) info
TZ time zone to generate new indexes at midnight none

How to use

Prerequisites

Before creating a docker container, a healthy instance of Elasticsearch service must be running.

Options
Key Default Value Required
elasticsearch-fields containerID,containerName,containerImageName,containerCreated no
elasticsearch-index docker-%Y.%m.%d no
elasticsearch-insecure false no
elasticsearch-password no no
elasticsearch-sniff yes no
elasticsearch-timeout 10s no
elasticsearch-type log no
elasticsearch-username no no
elasticsearch-url no yes
elasticsearch-version 5 no
elasticsearch-bulk-actions 100 no
elasticsearch-bulk-size 5242880 no
elasticsearch-bulk-flush-interval 5s no
elasticsearch-bulk-workers 1 no
grok-named-capture true no
grok-pattern no no
grok-pattern-from no no
grok-pattern-splitter and no
grok-pattern-match no no
elasticsearch-url
elasticsearch-insecure
  • insecure controls whether a client verifies the server's certificate chain and host name. If insecure is true, TLS accepts any certificate presented by the server and any host name in that certificate. In this mode, TLS is susceptible to man-in-the-middle attacks.
  • examples: 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False
elasticsearch-index
  • index to write log messages to
  • examples: docker, logging-%F, docker-%Y.%m.%d

Interpreted sequences are:

Regex Description
%b locale's abbreviated month name (Jan)
%B locale's full month name (January)
%d day of month (01)
%F full date; same as %Y.%m.%d
%j day of year (001..366)
%m month (01..12)
%y last two digits of year (00..99)
%Y year (2018)
elasticsearch-username
  • username to authenticate to a secure Elasticsearch cluster
  • example: elastic
elasticsearch-password
  • password to authenticate to a secure Elasticsearch cluster
  • WARNING: the password will be stored as clear text password in the container config. This will be changed in the future versions.
  • examples: changeme
elasticsearch-type
  • type to write log messages to
  • example: log
elasticsearch-timeout
  • timeout maximum time in seconds that a connection is allowed to take
  • examples: 300ms, 1s, 2h45m
elasticsearch-fields
  • fields to log to Elasticsearch Cluster
  • examples: containerID,containerLabels,containerEnv or none
elasticsearch-sniff
  • sniff uses the Node Info API to return the list of nodes in the cluster. It uses the list of URLs passed on startup plus the list of URLs found by the preceding sniffing process.
  • examples: 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False
elasticsearch-version
  • version of Elasticsearch cluster
  • examples: 1, 2, 5, 6
elasticsearch-bulk-actions
  • bulk-actions specifies when to flush based on the number of actions currently added
  • examples: 1000
elasticsearch-bulk-workers
  • bulk-workers is the number of concurrent workers allowed to be executed
  • examples: 2
elasticsearch-bulk-size
  • bulk-size specifies when to flush based on the size (in bytes) of the actions currently added. Set to -1 to disable it.
  • examples: 1024, -1
elasticsearch-bulk-flush-interval
  • bulk-flush-interval specifies when to flush at the end of the given interval
  • examples: 300ms, 1s, 2h45m
grok-pattern
  • pattern add customer pattern
  • examples: CUSTOM_IP=(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)
grok-pattern-from
  • pattern-from add custom pattern from a file or folder
  • examples: /srv/grok/pattern (this directory must be bound or linked inside the plugins's rootfs)
grok-pattern-splitter
  • pattern-splitter is used for splitting multiple patterns from grok-pattern
  • examples: " AND " (with white spaces before and after the word AND)
grok-match
  • match the log line to parse
  • examples: %{WORD:test1} %{WORD:test2}
grok-named-capture
  • named-capture parse each inner pattern or only named captures
  • examples: 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False

How to test

1 - Creating and running a container:

$ docker run --rm -ti \
    --log-driver elasticsearch \
    --log-opt elasticsearch-url=https://127.0.0.1:9200 \
    --log-opt elasticsearch-insecure=false \
    --log-opt elasticsearch-username=elastic \
    --log-opt elasticsearch-password=changeme \
    --log-opt elasticsearch-sniff=false \
    --log-opt elasticsearch-index=docker-%F \
    --log-opt elasticsearch-type=log \
    --log-opt elasticsearch-timeout=60s \
    --log-opt elasticsearch-version=5 \
    --log-opt elasticsearch-fields=containerID,containerName,containerImageID,containerImageName,containerCreated \
    --log-opt elasticsearch-bulk-workers=1 \
    --log-opt elasticsearch-bulk-actions=1000 \
    --log-opt elasticsearch-bulk-size=1024 \
    --log-opt elasticsearch-bulk-flush-interval=1s \
        alpine echo -n "this is a test logging message"

Search in Elasticsearch for the log message:

$ curl 127.0.0.1:9200/docker/log/_search\?pretty=true

{
  "_index" : "docker-2006.01.02",
  "_type" : "log",
  "_id" : "AWCywmj6Dipxk6-_e8T5",
  "_score" : 1.0,
  "_source" : {
    "containerID" : "f7d986496f66",
    "containerName" => "focused_lumiere",
    "containerImageID" : "sha256:8d254d3d0dca3e3ee8f377e752af11e0909b51133da614af4b30e4769aff5a44",
    "containerImageName" : "alpine",
    "containerCreated" : "2018-01-18T21:45:29.053364087Z",
    "source" : "stdout",
    "timestamp" : "2018-01-18T21:45:30.294363869Z",
    "partial" : false,
    "message" : "this is a test message"
  }
}

2 - Using grok extension for parsing the log messages:

docker run --rm -ti \
    --log-driver rchicoli/docker-log-elasticsearch:latest \
    --log-opt elasticsearch-url=https://127.0.0.1:9200 \
    --log-opt grok-match='%{COMMONAPACHELOG}' \
    --log-opt grok-named-capture=true \
    alpine echo -n "127.0.0.1 - - [23/Apr/2014:22:58:32 +0200] \"GET /index.php HTTP/1.1\" 404 $((RANDOM))"

The apache log line above will be displayed in elasticsearch as following:

"_source": {
  "containerID": "af7b2f782963",
  "containerName": "dazzling_knuth",
  "containerImageName": "alpine",
  "containerCreated": "2018-03-16T22:13:42.27376049Z",
  "source": "stdout",
  "timestamp": "2018-03-16T22:13:42.810856646Z",
  "partial": true,
  "grok": {
    "auth": "-",
    "bytes": "28453",
    "clientip": "127.0.0.1",
    "httpversion": "1.1",
    "ident": "-",
    "rawrequest": "",
    "request": "/index.php",
    "response": "404",
    "timestamp": "23/Apr/2014:22:58:32 +0200",
    "verb": "GET"
  }
}

In case you want to save the complete log line and its meta fields, you can set grok-named-capture to false, this will show some duplicated values though:

"grok": {
  "BASE10NUM": "16406",
  "COMMONAPACHELOG": "127.0.0.1 - - [23/Apr/2014:22:58:32 +0200] \"GET /index.php HTTP/1.1\" 404 16406",
  "EMAILADDRESS": "",
  "EMAILLOCALPART": "",
  "HOSTNAME": "",
  "HOUR": "22",
  "INT": "+0200",
  "IP": "127.0.0.1",
  "IPV4": "127.0.0.1",
  "IPV6": "",
  "MINUTE": "58",
  "MONTH": "Apr",
  "MONTHDAY": "23",
  "SECOND": "32",
  "TIME": "22:58:32",
  "USER": "-",
  "USERNAME": "-",
  "YEAR": "2014",
  "auth": "-",
  "bytes": "16406",
  "clientip": "127.0.0.1",
  "httpversion": "1.1",
  "ident": "-",
  "rawrequest": "",
  "request": "/index.php",
  "response": "404",
  "timestamp": "23/Apr/2014:22:58:32 +0200",
  "verb": "GET"
}

3 - There are two different ways of adding custom grok patterns.

a. by providing the grok pattern as parameter, e.g.:

docker run --rm -ti --log-opt grok-named-capture=false \
  --log-driver rchicoli/docker-log-elasticsearch:development \
  --log-opt elasticsearch-url=http://172.31.0.2:9200 \
  --log-opt grok-pattern='MY_NUMBER=(?:[+-]?(?:[0-9]+)) && MY_USER=[a-zA-Z0-9._-]+ && MY_PATTERN=%{MY_NUMBER:random_number} %{MY_USER:user}' \
  --log-opt grok-pattern-splitter=' && ' \
  --log-opt grok-match='%{MY_PATTERN:log}' \
  alpine echo -n "$((RANDOM)) tester"

Note we are using multiple custom patterns and also a different pattern splitter.

"_source": {
  "containerID": "e11b45c911b4",
  "containerName": "ecstatic_leakey",
  "containerImageName": "alpine",
  "containerCreated": "2018-03-16T22:47:39.39245724Z",
  "source": "stdout",
  "timestamp": "2018-03-16T22:47:39.940595233Z",
  "partial": true,
  "grok": {
    "log": "2921 tester",
    "random_number": "2921",
    "user": "tester"
  }
}

b. by providing a directory with different grok patterns in it or just a single file, e.g.:

At first, you have to place the file or directory inside the docker's rootfs. It is up to you to choose the right way to do it. You could link the file, mount the directory or simply copy it inside /var/lib/docker/plugin/<plugin-id>/rootfs/. Afterwards you can pass it as following:

docker run -ti --rm --log-driver rchicoli/docker-log-elasticsearch:development --log-opt elasticsearch-url=http://172.31.0.2:9200 \
  --log-opt grok-pattern-from="/patterns" \
  rchicoli/webapper

4 - If grok is not able to parse the log line, then it will still send the unparsed message to Elasticsearch with an error description, e.g.:

"grok": {
  "err": "grok pattern does not match line",
  "line": "4721 tester"
}

Limitations

There are some limitations so far, which will be improved at some point.

  • grok parses everything to a string field, convertion type is only possible if a template has been added to elasticsearch
  • grok-pattern-from requires the file to be inside the plugin's rootfs. Alternative is the plugin's mount source.

Description of fields

Static Fields are always present

Field Description Default
message The log message itself yes
source Source of the log message as reported by docker yes
timestamp Timestamp that the log was collected by the log driver yes
partial Whether docker reported that the log message was only partially collected yes

Dynamic Fields: can be provided by elasticsearch-fields log paramenter

Field Description Default
config Config provided by log-opt no
containerID Id of the container that generated the log message yes
containerName Name of the container that generated the log message yes
containerArgs Arguments of the container entrypoint no
containerImageID ID of the container's image no
containerImageName Name of the container's image yes
containerCreated Timestamp of the container's creation yes
containerEnv Environment of the container no
containerLabels Label of the container no
containerLogPath Path of the container's Log no
daemonName Name of the container's daemon no