Skip to content
This repository has been archived by the owner on Jun 25, 2019. It is now read-only.

Unable to register in trackit #69

Open
sathyajackson opened this issue Dec 6, 2018 · 35 comments
Open

Unable to register in trackit #69

sathyajackson opened this issue Dec 6, 2018 · 35 comments
Assignees
@thibautcornolti

Comments

@sathyajackson
Copy link

I have set up the complete UI using the docker and its up and running. Now that I am trying to register the same but unable to.

when I try to register, I get an error as "An error has occurred".

Not sure what wrong can someone help me out to get the UI up and running.
@thibautcornolti thibautcornolti self-assigned this Dec 6, 2018
@thibautcornolti
Copy link
Member

Hi @sathyajackson

Can I have some logs? You can get them by typing docker-compose logs -f.

Thanks.

@lfrancois
Copy link
Member

@sathyajackson can we setup a call with you later this week or next week? So that we can troubleshoot the issue with you?

@sathyajackson
Copy link
Author

Sorry sir, if your planning to make me buy then I am not the right person for you talk....

Even the company I work are not ready to buy paid software sorry........

@sathyajackson
Copy link
Author

Not my fault......peace

@lfrancois
Copy link
Member

No worries :) We just want to understand the issue and get constructive feedback about what we are building. You can reach me at [email protected] and we will setup a troubleshooting session. Also you can test the free version on https://trackit.io. Nobody will ask you any credit card or anything :)

@sathyajackson
Copy link
Author

Here is the complete log file

WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
Attaching to trackit_api_1_d0ea38fddddd, trackit_es_1_c00afa428f94, trackit_sql_1_baad39f9a608, trackit_ui_1_c7e00a815717
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:32.478736607Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.3:3306: getsockopt: connection refused"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:34.479610691Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.3:3306: getsockopt: connection refused"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:36.480478594Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.3:3306: getsockopt: connection refused"}
es_1_c00afa428f94 | [2018-12-06T14:43:33,114][INFO ][o.e.n.Node ] [] initializing ...
sql_1_baad39f9a608 | Initializing database
es_1_c00afa428f94 | [2018-12-06T14:43:33,166][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] using [1] data paths, mounts [[/ (overlay)]], net usable_space [8.4gb], net total_space [29.4gb], spins? [possibly], types [overlay]
sql_1_baad39f9a608 |
sql_1_baad39f9a608 | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:38.481242486Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.3:3306: getsockopt: connection refused"}
sql_1_baad39f9a608 | To do so, start the server, then issue the following commands:
es_1_c00afa428f94 | [2018-12-06T14:43:33,166][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] heap size [1.9gb], compressed ordinary object pointers [true]
api_1_d0ea38fddddd | {"level":"info","time":"2018-12-06T14:43:40.48230592Z","message":"Successfully connected to SQL database."}
ui_1_c7e00a815717 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
sql_1_baad39f9a608 |
es_1_c00afa428f94 | [2018-12-06T14:43:33,167][INFO ][o.e.n.Node ] node name [Izt4zdP] derived from node ID [Izt4zdPRTW2IHvngLppy2w]; set [node.name] to override
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:46.249540739Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
ui_1_c7e00a815717 | 'Supervisord is running as root and it is searching '
sql_1_baad39f9a608 | '/usr/bin/mysqladmin' -u root password 'new-password'
es_1_c00afa428f94 | [2018-12-06T14:43:33,168][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:43:53.888146148Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | '/usr/bin/mysqladmin' -u root -h password 'new-password'
ui_1_c7e00a815717 | 2018-12-06 14:44:48,247 CRIT Supervisor running as root (no user in config file)
es_1_c00afa428f94 | [2018-12-06T14:43:33,168][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:01.356401632Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 |
ui_1_c7e00a815717 | 2018-12-06 14:44:48,248 INFO supervisord started with pid 1
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [aggs-matrix-stats]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:08.966041141Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | Alternatively you can run:
ui_1_c7e00a815717 | 2018-12-06 14:44:49,250 INFO spawned: 'webui' with pid 10
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [ingest-common]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:16.532639788Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | '/usr/bin/mysql_secure_installation'
ui_1_c7e00a815717 | 2018-12-06 14:44:50,428 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-expression]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:24.19991604Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 |
ui_1_c7e00a815717 | 2018-12-06 15:30:52,465 WARN received SIGTERM indicating exit request
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-groovy]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:31.702865478Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | which will also give you the option of removing the test
ui_1_c7e00a815717 | 2018-12-06 15:30:52,466 INFO waiting for webui to die
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:39.203860775Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-mustache]
sql_1_baad39f9a608 | databases and anonymous user created by default. This is
ui_1_c7e00a815717 | 2018-12-06 15:30:52,467 INFO stopped: webui (terminated by SIGTERM)
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-painless]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:46.748528399Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | strongly recommended for production servers.
ui_1_c7e00a815717 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [parent-join]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:44:53.752770785Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"}
sql_1_baad39f9a608 |
ui_1_c7e00a815717 | 'Supervisord is running as root and it is searching '
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [percolator]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:45:00.947007596Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | See the MariaDB Knowledgebase at http://mariadb.com/kb or the
ui_1_c7e00a815717 | 2018-12-06 15:32:40,828 CRIT Supervisor running as root (no user in config file)
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [reindex]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:45:08.372114485Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | MySQL manual for more instructions.
ui_1_c7e00a815717 | 2018-12-06 15:32:40,830 INFO supervisord started with pid 1
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty3]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:45:15.532798053Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 |
ui_1_c7e00a815717 | 2018-12-06 15:32:41,831 INFO spawned: 'webui' with pid 10
es_1_c00afa428f94 | [2018-12-06T14:43:34,347][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty4]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:45:22.812170558Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | Please report any problems at http://mariadb.org/jira
ui_1_c7e00a815717 | 2018-12-06 15:32:43,025 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
es_1_c00afa428f94 | [2018-12-06T14:43:34,348][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-geoip]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T14:45:30.300248305Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 |
ui_1_c7e00a815717 | 2018-12-07 13:42:52,609 WARN received SIGTERM indicating exit request
es_1_c00afa428f94 | [2018-12-06T14:43:34,348][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-user-agent]
api_1_d0ea38fddddd | {"level":"error","time":"2018-12-06T14:45:32.300391772Z","message":"Failed to connect to ElasticSearch database. Not retrying."}
sql_1_baad39f9a608 | The latest information about MariaDB is available at http://mariadb.org/.
es_1_c00afa428f94 | [2018-12-06T14:43:34,348][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [x-pack]
api_1_d0ea38fddddd | panic: runtime error: invalid memory address or nil pointer dereference
es_1_c00afa428f94 | [2018-12-06T14:43:37,156][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/56] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
sql_1_baad39f9a608 | You can find additional information about the MySQL part at:
ui_1_c7e00a815717 | 2018-12-07 13:42:52,610 INFO waiting for webui to die
es_1_c00afa428f94 | [2018-12-06T14:43:37,188][INFO ][o.e.d.DiscoveryModule ] [Izt4zdP] using discovery type [zen]
api_1_d0ea38fddddd | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a0f33]
sql_1_baad39f9a608 | http://dev.mysql.com
ui_1_c7e00a815717 | 2018-12-07 13:42:52,611 INFO stopped: webui (terminated by SIGTERM)
es_1_c00afa428f94 | [2018-12-06T14:43:37,800][INFO ][o.e.n.Node ] initialized
api_1_d0ea38fddddd |
sql_1_baad39f9a608 | Consider joining MariaDB's strong and vibrant community:
ui_1_c7e00a815717 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
es_1_c00afa428f94 | [2018-12-06T14:43:37,800][INFO ][o.e.n.Node ] [Izt4zdP] starting ...
api_1_d0ea38fddddd | goroutine 1 [running]:
sql_1_baad39f9a608 | https://mariadb.org/get-involved/
ui_1_c7e00a815717 | 'Supervisord is running as root and it is searching '
es_1_c00afa428f94 | [2018-12-06T14:43:37,993][INFO ][o.e.t.TransportService ] [Izt4zdP] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300}
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc4204c5980, 0x108a829, 0x3, 0xc4204ab9e0, 0x14, 0xc42054a0c0, 0xe06aa0, 0xc4203f6dd0, ...)
sql_1_baad39f9a608 |
ui_1_c7e00a815717 | 2018-12-07 13:44:41,637 CRIT Supervisor running as root (no user in config file)
es_1_c00afa428f94 | [2018-12-06T14:43:38,001][INFO ][o.e.b.BootstrapChecks ] [Izt4zdP] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
sql_1_baad39f9a608 | Database initialized
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3
ui_1_c7e00a815717 | 2018-12-07 13:44:41,639 INFO supervisord started with pid 1
es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed
sql_1_baad39f9a608 | MySQL init process in progress...
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequest(0x0, 0x189d380, 0xc4204c5980, 0x108a829, 0x3, 0xc4204ab9e0, 0x14, 0xc42054a0c0, 0xe06aa0, 0xc4203f6dd0, ...)
ui_1_c7e00a815717 | 2018-12-07 13:44:42,641 INFO spawned: 'webui' with pid 9
es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+mariajessie) starting as process 102 ...
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f
es_1_c00afa428f94 | [2018-12-06T14:43:38,009][INFO ][o.e.n.Node ] [Izt4zdP] stopping ...
ui_1_c7e00a815717 | 2018-12-07 13:44:43,845 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc42056fe38, 0x189d380, 0xc4204c5980, 0x189d380, 0xc4204c5980, 0xc4203f6d90)
es_1_c00afa428f94 | [2018-12-06T14:43:38,115][INFO ][o.e.n.Node ] [Izt4zdP] stopped
ui_1_c7e00a815717 | 2018-12-07 13:49:10,098 WARN received SIGTERM indicating exit request
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Uses event mutexes
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111
es_1_c00afa428f94 | [2018-12-06T14:43:38,115][INFO ][o.e.n.Node ] [Izt4zdP] closing ...
ui_1_c7e00a815717 | 2018-12-07 13:49:10,098 INFO waiting for webui to die
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Compressed tables use zlib 1.2.8
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1()
es_1_c00afa428f94 | [2018-12-06T14:43:38,123][INFO ][o.e.n.Node ] [Izt4zdP] closed
ui_1_c7e00a815717 | 2018-12-07 13:49:10,100 INFO stopped: webui (terminated by SIGTERM)
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Using Linux native AIO
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea
es_1_c00afa428f94 | [2018-12-06T14:43:38,126][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
ui_1_c7e00a815717 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Number of pools: 1
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init()
es_1_c00afa428f94 | [2018-12-06T14:44:49,802][INFO ][o.e.n.Node ] [] initializing ...
ui_1_c7e00a815717 | 'Supervisord is running as root and it is searching '
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Using SSE2 crc32 instructions
api_1_d0ea38fddddd | :1 +0x183
es_1_c00afa428f94 | [2018-12-06T14:44:49,861][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] using [1] data paths, mounts [[/ (overlay)]], net usable_space [7.9gb], net total_space [29.4gb], spins? [possibly], types [overlay]
ui_1_c7e00a815717 | 2018-12-07 13:49:11,308 CRIT Supervisor running as root (no user in config file)
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init()
es_1_c00afa428f94 | [2018-12-06T14:44:49,861][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] heap size [1.9gb], compressed ordinary object pointers [true]
ui_1_c7e00a815717 | 2018-12-07 13:49:11,309 INFO supervisord started with pid 1
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Completed initialization of buffer pool
es_1_c00afa428f94 | [2018-12-06T14:44:49,863][INFO ][o.e.n.Node ] node name [Izt4zdP] derived from node ID [Izt4zdPRTW2IHvngLppy2w]; set [node.name] to override
api_1_d0ea38fddddd | :1 +0x8c
ui_1_c7e00a815717 | 2018-12-07 13:49:12,311 INFO spawned: 'webui' with pid 10
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064142829312 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
es_1_c00afa428f94 | [2018-12-06T14:44:49,863][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
api_1_d0ea38fddddd | main.init()
ui_1_c7e00a815717 | 2018-12-07 13:49:13,446 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Highest supported file format is Barracuda.
es_1_c00afa428f94 | [2018-12-06T14:44:49,863][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
api_1_d0ea38fddddd | :1 +0x78
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: 128 out of 128 rollback segments are active.
es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [aggs-matrix-stats]
api_1_d0ea38fddddd | {"level":"info","time":"2018-12-06T15:32:41.979048074Z","message":"Successfully connected to SQL database."}
es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [ingest-common]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:32:47.083580545Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Creating shared tablespace for temporary tables
es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-expression]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:32:54.258577463Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-groovy]
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:01.683415811Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-mustache]
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: Waiting for purge to start
es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-painless]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:08.828629445Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] InnoDB: 5.7.19 started; log sequence number 1619987
es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [parent-join]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:16.423000405Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140063765681920 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [percolator]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:23.733119339Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140063765681920 [Note] InnoDB: Buffer pool(s) load completed at 181206 14:43:34
es_1_c00afa428f94 | [2018-12-06T14:44:50,894][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [reindex]
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] Plugin 'FEEDBACK' is disabled.
es_1_c00afa428f94 | [2018-12-06T14:44:50,895][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty3]
es_1_c00afa428f94 | [2018-12-06T14:44:50,895][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty4]
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Warning] 'user' entry 'root@397e1235ec88' ignored in --skip-name-resolve mode.
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:30.919414391Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T14:44:50,895][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-geoip]
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode.
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:38.10388314Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T14:44:50,895][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-user-agent]
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] Reading of all Master_info entries succeded
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:45.418916162Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T14:44:50,895][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [x-pack]
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] Added new Master_info '' to hash table
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:52.590513122Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T14:44:53,377][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/55] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
sql_1_baad39f9a608 | 2018-12-06 14:43:34 140064864130944 [Note] mysqld: ready for connections.
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:33:59.853203755Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T14:44:53,412][INFO ][o.e.d.DiscoveryModule ] [Izt4zdP] using discovery type [zen]
sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+mariajessie' socket: '/var/run/mysqld/mysqld.sock' port: 0 mariadb.org binary distribution
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:34:06.933498739Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T14:44:54,034][INFO ][o.e.n.Node ] initialized
sql_1_baad39f9a608 | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:34:14.122299203Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | GENERATED ROOT PASSWORD: Uengoo6zoov4loof0ji2hies1ga0ain9
es_1_c00afa428f94 | [2018-12-06T14:44:54,035][INFO ][o.e.n.Node ] [Izt4zdP] starting ...
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:34:21.199974774Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:36 140064722687744 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode.
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-06T15:34:28.422354273Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:36 140064722687744 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode.
es_1_c00afa428f94 | [2018-12-06T14:44:54,225][INFO ][o.e.t.TransportService ] [Izt4zdP] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300}
api_1_d0ea38fddddd | {"level":"error","time":"2018-12-06T15:34:30.42248925Z","message":"Failed to connect to ElasticSearch database. Not retrying."}
sql_1_baad39f9a608 |
es_1_c00afa428f94 | [2018-12-06T14:44:54,233][INFO ][o.e.b.BootstrapChecks ] [Izt4zdP] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
api_1_d0ea38fddddd | panic: runtime error: invalid memory address or nil pointer dereference
sql_1_baad39f9a608 | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/schema.sql
es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed
api_1_d0ea38fddddd | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a0f33]
sql_1_baad39f9a608 |
es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
sql_1_baad39f9a608 |
es_1_c00afa428f94 | [2018-12-06T14:44:54,241][INFO ][o.e.n.Node ] [Izt4zdP] stopping ...
sql_1_baad39f9a608 | 2018-12-06 14:43:37 140064722384640 [Note] mysqld (unknown): Normal shutdown
api_1_d0ea38fddddd |
es_1_c00afa428f94 | [2018-12-06T14:44:54,348][INFO ][o.e.n.Node ] [Izt4zdP] stopped
sql_1_baad39f9a608 |
api_1_d0ea38fddddd | goroutine 1 [running]:
es_1_c00afa428f94 | [2018-12-06T14:44:54,349][INFO ][o.e.n.Node ] [Izt4zdP] closing ...
sql_1_baad39f9a608 | 2018-12-06 14:43:37 140064722384640 [Note] Event Scheduler: Purging the queue. 0 events
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc4205d41e0, 0x108a829, 0x3, 0xc420503a80, 0x14, 0xc4205abe90, 0xe06aa0, 0xc42042ac80, ...)
es_1_c00afa428f94 | [2018-12-06T14:44:54,357][INFO ][o.e.n.Node ] [Izt4zdP] closed
sql_1_baad39f9a608 | 2018-12-06 14:43:37 140064017331968 [Note] InnoDB: FTS optimize thread exiting.
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3
es_1_c00afa428f94 | [2018-12-06T15:32:42,634][INFO ][o.e.n.Node ] [] initializing ...
sql_1_baad39f9a608 | 2018-12-06 14:43:37 140064722384640 [Note] InnoDB: Starting shutdown...
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequest(0x0, 0x189d380, 0xc4205d41e0, 0x108a829, 0x3, 0xc420503a80, 0x14, 0xc4205abe90, 0xe06aa0, 0xc42042ac80, ...)
es_1_c00afa428f94 | [2018-12-06T15:32:42,689][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] using [1] data paths, mounts [[/ (overlay)]], net usable_space [7.6gb], net total_space [29.4gb], spins? [possibly], types [overlay]
sql_1_baad39f9a608 | 2018-12-06 14:43:37 140063765681920 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f
sql_1_baad39f9a608 | 2018-12-06 14:43:37 140063765681920 [Note] InnoDB: Buffer pool(s) dump completed at 181206 14:43:37
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc420601e38, 0x189d380, 0xc4205d41e0, 0x189d380, 0xc4205d41e0, 0xc42042ac40)
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140064722384640 [Note] InnoDB: Shutdown completed; log sequence number 2179604
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111
es_1_c00afa428f94 | [2018-12-06T15:32:42,689][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] heap size [1.9gb], compressed ordinary object pointers [true]
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140064722384640 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1()
es_1_c00afa428f94 | [2018-12-06T15:32:42,690][INFO ][o.e.n.Node ] node name [Izt4zdP] derived from node ID [Izt4zdPRTW2IHvngLppy2w]; set [node.name] to override
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140064722384640 [Note] mysqld: Shutdown complete
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea
sql_1_baad39f9a608 |
es_1_c00afa428f94 | [2018-12-06T15:32:42,691][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
es_1_c00afa428f94 | [2018-12-06T15:32:42,691][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
es_1_c00afa428f94 | [2018-12-06T15:32:43,741][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [aggs-matrix-stats]
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init()
es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [ingest-common]
api_1_d0ea38fddddd | :1 +0x183
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init()
sql_1_baad39f9a608 |
es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-expression]
api_1_d0ea38fddddd | :1 +0x8c
sql_1_baad39f9a608 | MySQL init process done. Ready for start up.
es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-groovy]
api_1_d0ea38fddddd | main.init()
es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-mustache]
api_1_d0ea38fddddd | :1 +0x78
es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-painless]
api_1_d0ea38fddddd | {"level":"info","time":"2018-12-07T13:44:42.762834853Z","message":"Successfully connected to SQL database."}
sql_1_baad39f9a608 |
es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [parent-join]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:44:47.855187737Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+mariajessie) starting as process 1 ...
es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [percolator]
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:44:55.204535439Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [reindex]
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Uses event mutexes
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:02.530741589Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty3]
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Compressed tables use zlib 1.2.8
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:10.156413673Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T15:32:43,742][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty4]
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Using Linux native AIO
es_1_c00afa428f94 | [2018-12-06T15:32:43,743][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-geoip]
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Number of pools: 1
es_1_c00afa428f94 | [2018-12-06T15:32:43,743][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-user-agent]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:17.543638008Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Using SSE2 crc32 instructions
es_1_c00afa428f94 | [2018-12-06T15:32:43,743][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [x-pack]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:25.198023378Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
es_1_c00afa428f94 | [2018-12-06T15:32:46,221][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/56] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:32.683421179Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Completed initialization of buffer pool
es_1_c00afa428f94 | [2018-12-06T15:32:46,247][INFO ][o.e.d.DiscoveryModule ] [Izt4zdP] using discovery type [zen]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:40.051306241Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140549769058048 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
es_1_c00afa428f94 | [2018-12-06T15:32:46,746][INFO ][o.e.n.Node ] initialized
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:47.468262623Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T15:32:46,746][INFO ][o.e.n.Node ] [Izt4zdP] starting ...
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Highest supported file format is Barracuda.
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:45:54.697136828Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T15:32:46,912][INFO ][o.e.t.TransportService ] [Izt4zdP] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300}
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: 128 out of 128 rollback segments are active.
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:46:01.864115713Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T15:32:46,918][INFO ][o.e.b.BootstrapChecks ] [Izt4zdP] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Creating shared tablespace for temporary tables
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:46:09.181044801Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:46:16.538487676Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:46:23.730707588Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T15:32:46,926][INFO ][o.e.n.Node ] [Izt4zdP] stopping ...
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] InnoDB: 5.7.19 started; log sequence number 2179604
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T13:46:31.146089002Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-06T15:32:47,019][INFO ][o.e.n.Node ] [Izt4zdP] stopped
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140549600286464 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
api_1_d0ea38fddddd | {"level":"error","time":"2018-12-07T13:46:33.146250193Z","message":"Failed to connect to ElasticSearch database. Not retrying."}
es_1_c00afa428f94 | [2018-12-06T15:32:47,019][INFO ][o.e.n.Node ] [Izt4zdP] closing ...
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] Plugin 'FEEDBACK' is disabled.
api_1_d0ea38fddddd | panic: runtime error: invalid memory address or nil pointer dereference
es_1_c00afa428f94 | [2018-12-06T15:32:47,027][INFO ][o.e.n.Node ] [Izt4zdP] closed
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] Server socket created on IP: '::'.
api_1_d0ea38fddddd | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a0f33]
es_1_c00afa428f94 | [2018-12-06T15:32:47,028][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode.
api_1_d0ea38fddddd |
es_1_c00afa428f94 | [2018-12-07T13:44:43,394][INFO ][o.e.n.Node ] [] initializing ...
api_1_d0ea38fddddd | goroutine 1 [running]:
es_1_c00afa428f94 | [2018-12-07T13:44:43,454][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] using [1] data paths, mounts [[/ (overlay)]], net usable_space [7.2gb], net total_space [29.4gb], spins? [possibly], types [overlay]
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] Reading of all Master_info entries succeded
es_1_c00afa428f94 | [2018-12-07T13:44:43,454][INFO ][o.e.e.NodeEnvironment ] [Izt4zdP] heap size [1.9gb], compressed ordinary object pointers [true]
es_1_c00afa428f94 | [2018-12-07T13:44:43,455][INFO ][o.e.n.Node ] node name [Izt4zdP] derived from node ID [Izt4zdPRTW2IHvngLppy2w]; set [node.name] to override
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] Added new Master_info '' to hash table
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc4205934a0, 0x108a829, 0x3, 0xc42041d720, 0x14, 0xc42051e630, 0xe06aa0, 0xc4204411e0, ...)
es_1_c00afa428f94 | [2018-12-07T13:44:43,455][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140550491084672 [Note] mysqld: ready for connections.
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3
es_1_c00afa428f94 | [2018-12-07T13:44:43,455][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+mariajessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequest(0x0, 0x189d380, 0xc4205934a0, 0x108a829, 0x3, 0xc42041d720, 0x14, 0xc42051e630, 0xe06aa0, 0xc4204411e0, ...)
es_1_c00afa428f94 | [2018-12-07T13:44:44,605][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [aggs-matrix-stats]
sql_1_baad39f9a608 | 2018-12-06 14:43:38 140549600286464 [Note] InnoDB: Buffer pool(s) load completed at 181206 14:43:38
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f
es_1_c00afa428f94 | [2018-12-07T13:44:44,605][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [ingest-common]
sql_1_baad39f9a608 | 2018-12-06 14:45:32 140550325155584 [Warning] Aborted connection 8 to db: 'trackit' user: 'trackit' host: '10.19.84.2' (Got an error reading communication packets)
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc420543e38, 0x189d380, 0xc4205934a0, 0x189d380, 0xc4205934a0, 0xc4204411a0)
es_1_c00afa428f94 | [2018-12-07T13:44:44,605][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-expression]
sql_1_baad39f9a608 | 2018-12-06 15:30:52 140550325155584 [Note] mysqld (unknown): Normal shutdown
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111
es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-groovy]
sql_1_baad39f9a608 |
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1()
es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-mustache]
sql_1_baad39f9a608 | 2018-12-06 15:30:52 140550325155584 [Note] Event Scheduler: Purging the queue. 0 events
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea
es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [lang-painless]
sql_1_baad39f9a608 | 2018-12-06 15:30:52 140549642249984 [Note] InnoDB: FTS optimize thread exiting.
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init()
es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [parent-join]
sql_1_baad39f9a608 | 2018-12-06 15:30:52 140550325155584 [Note] InnoDB: Starting shutdown...
api_1_d0ea38fddddd | :1 +0x183
es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [percolator]
sql_1_baad39f9a608 | 2018-12-06 15:30:52 140549600286464 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init()
es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [reindex]
sql_1_baad39f9a608 | 2018-12-06 15:30:52 140549600286464 [Note] InnoDB: Buffer pool(s) dump completed at 181206 15:30:52
api_1_d0ea38fddddd | :1 +0x8c
es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty3]
sql_1_baad39f9a608 | 2018-12-06 15:30:53 140550325155584 [Note] InnoDB: Shutdown completed; log sequence number 2179632
api_1_d0ea38fddddd | main.init()
api_1_d0ea38fddddd | :1 +0x78
sql_1_baad39f9a608 | 2018-12-06 15:30:53 140550325155584 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
es_1_c00afa428f94 | [2018-12-07T13:44:44,606][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded module [transport-netty4]
sql_1_baad39f9a608 | 2018-12-06 15:30:53 140550325155584 [Note] mysqld: Shutdown complete
sql_1_baad39f9a608 |
es_1_c00afa428f94 | [2018-12-07T13:44:44,607][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-geoip]
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+mariajessie) starting as process 1 ...
es_1_c00afa428f94 | [2018-12-07T13:44:44,607][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [ingest-user-agent]
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
es_1_c00afa428f94 | [2018-12-07T13:44:44,607][INFO ][o.e.p.PluginsService ] [Izt4zdP] loaded plugin [x-pack]
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Uses event mutexes
es_1_c00afa428f94 | [2018-12-07T13:44:47,140][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/56] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Compressed tables use zlib 1.2.8
es_1_c00afa428f94 | [2018-12-07T13:44:47,177][INFO ][o.e.d.DiscoveryModule ] [Izt4zdP] using discovery type [zen]
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Using Linux native AIO
es_1_c00afa428f94 | [2018-12-07T13:44:47,781][INFO ][o.e.n.Node ] initialized
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Number of pools: 1
es_1_c00afa428f94 | [2018-12-07T13:44:47,781][INFO ][o.e.n.Node ] [Izt4zdP] starting ...
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Using SSE2 crc32 instructions
es_1_c00afa428f94 | [2018-12-07T13:44:47,949][INFO ][o.e.t.TransportService ] [Izt4zdP] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300}
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
es_1_c00afa428f94 | [2018-12-07T13:44:47,955][INFO ][o.e.b.BootstrapChecks ] [Izt4zdP] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Completed initialization of buffer pool
es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139756584908544 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Highest supported file format is Barracuda.
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: 128 out of 128 rollback segments are active.
es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Creating shared tablespace for temporary tables
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
es_1_c00afa428f94 | [2018-12-07T13:44:47,962][INFO ][o.e.n.Node ] [Izt4zdP] stopping ...
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
trackit_api_1_d0ea38fddddd exited with code 2
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] InnoDB: 5.7.19 started; log sequence number 2179632
es_1_c00afa428f94 | [2018-12-07T13:44:48,048][INFO ][o.e.n.Node ] [Izt4zdP] stopped
es_1_c00afa428f94 | [2018-12-07T13:44:48,048][INFO ][o.e.n.Node ] [Izt4zdP] closing ...
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139756012824320 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
es_1_c00afa428f94 | [2018-12-07T13:44:48,055][INFO ][o.e.n.Node ] [Izt4zdP] closed
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] Plugin 'FEEDBACK' is disabled.
es_1_c00afa428f94 | [2018-12-07T13:44:48,057][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] Server socket created on IP: '::'.
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode.
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] Reading of all Master_info entries succeded
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] Added new Master_info '' to hash table
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139757308848000 [Note] mysqld: ready for connections.
sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+mariajessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
sql_1_baad39f9a608 | 2018-12-06 15:32:41 139756012824320 [Note] InnoDB: Buffer pool(s) load completed at 181206 15:32:41
sql_1_baad39f9a608 | 2018-12-06 15:34:30 139757167404800 [Warning] Aborted connection 8 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets)
sql_1_baad39f9a608 | 2018-12-07 13:42:52 139757167404800 [Note] mysqld (unknown): Normal shutdown
sql_1_baad39f9a608 |
sql_1_baad39f9a608 | 2018-12-07 13:42:52 139757167404800 [Note] Event Scheduler: Purging the queue. 0 events
sql_1_baad39f9a608 | 2018-12-07 13:42:52 139756465800960 [Note] InnoDB: FTS optimize thread exiting.
sql_1_baad39f9a608 | 2018-12-07 13:42:52 139757167404800 [Note] InnoDB: Starting shutdown...
sql_1_baad39f9a608 | 2018-12-07 13:42:52 139756012824320 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
sql_1_baad39f9a608 | 2018-12-07 13:42:52 139756012824320 [Note] InnoDB: Buffer pool(s) dump completed at 181207 13:42:52
sql_1_baad39f9a608 | 2018-12-07 13:42:54 139757167404800 [Note] InnoDB: Shutdown completed; log sequence number 2179660
sql_1_baad39f9a608 | 2018-12-07 13:42:54 139757167404800 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
sql_1_baad39f9a608 | 2018-12-07 13:42:54 139757167404800 [Note] mysqld: Shutdown complete
sql_1_baad39f9a608 |
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+mariajessie) starting as process 1 ...
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Uses event mutexes
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Compressed tables use zlib 1.2.8
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Using Linux native AIO
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Number of pools: 1
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Using SSE2 crc32 instructions
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Completed initialization of buffer pool
trackit_es_1_c00afa428f94 exited with code 78
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436050282240 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Highest supported file format is Barracuda.
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: 128 out of 128 rollback segments are active.
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Creating shared tablespace for temporary tables
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] InnoDB: 5.7.19 started; log sequence number 2179660
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140435674613504 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] Plugin 'FEEDBACK' is disabled.
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] Server socket created on IP: '::'.
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode.
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] Reading of all Master_info entries succeded
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] Added new Master_info '' to hash table
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140436771628928 [Note] mysqld: ready for connections.
sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+mariajessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
sql_1_baad39f9a608 | 2018-12-07 13:44:42 140435674613504 [Note] InnoDB: Buffer pool(s) load completed at 181207 13:44:42
sql_1_baad39f9a608 | 2018-12-07 13:46:33 140436136883968 [Warning] Aborted connection 8 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets)
sql_1_baad39f9a608 | 2018-12-07 13:49:07 140436136580864 [Note] mysqld (unknown): Normal shutdown
sql_1_baad39f9a608 |
sql_1_baad39f9a608 | 2018-12-07 13:49:07 140436136580864 [Note] Event Scheduler: Purging the queue. 0 events
sql_1_baad39f9a608 | 2018-12-07 13:49:07 140435926263552 [Note] InnoDB: FTS optimize thread exiting.
sql_1_baad39f9a608 | 2018-12-07 13:49:07 140436136580864 [Note] InnoDB: Starting shutdown...
sql_1_baad39f9a608 | 2018-12-07 13:49:07 140435674613504 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
sql_1_baad39f9a608 | 2018-12-07 13:49:07 140435674613504 [Note] InnoDB: Buffer pool(s) dump completed at 181207 13:49:07
sql_1_baad39f9a608 | 2018-12-07 13:49:09 140436136580864 [Note] InnoDB: Shutdown completed; log sequence number 2179688
sql_1_baad39f9a608 | 2018-12-07 13:49:09 140436136580864 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
sql_1_baad39f9a608 | 2018-12-07 13:49:09 140436136580864 [Note] mysqld: Shutdown complete
sql_1_baad39f9a608 |
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+mariajessie) starting as process 1 ...
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Uses event mutexes
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Compressed tables use zlib 1.2.8
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Using Linux native AIO
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Number of pools: 1
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Using SSE2 crc32 instructions
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Completed initialization of buffer pool
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140542484383488 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
sql_1_baad39f9a608 | 2018-12-07 13:49:10 140543205656448 [Note] InnoDB: Highest supported file format is Barracuda.
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] InnoDB: 128 out of 128 rollback segments are active.
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] InnoDB: Creating shared tablespace for temporary tables
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] InnoDB: 5.7.19 started; log sequence number 2179688
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140542243489536 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] Plugin 'FEEDBACK' is disabled.
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] Server socket created on IP: '::'.
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode.
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] Reading of all Master_info entries succeded
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] Added new Master_info '' to hash table
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140543205656448 [Note] mysqld: ready for connections.
sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+mariajessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
sql_1_baad39f9a608 | 2018-12-07 13:49:11 140542243489536 [Note] InnoDB: Buffer pool(s) load completed at 181207 13:49:11

@thibautcornolti
Copy link
Member

Your error seems to be: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

According to elastic.co, you should try sudo ulimit -n 65536.

@sathyajackson
Copy link
Author

I tired but the still the same. Please find the logs below.

docker-compose logs -f
WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
Attaching to trackit_api_1_d0ea38fddddd, trackit_es_1_c00afa428f94, trackit_sql_1_baad39f9a608, trackit_ui_1_c7e00a815717
api_1_d0ea38fddddd | {"level":"info","time":"2018-12-07T15:33:45.49718687Z","message":"Successfully connected to SQL database."}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:33:50.58063346Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:33:57.9534805Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:05.594416443Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:13.105674314Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-07T15:33:46,062][INFO ][o.e.n.Node ] [] initializing ...
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:20.756037608Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-07T15:33:46,128][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] using [1] data paths, mounts [[/ (overlay)]], net usable_space [5.3gb], net total_space [29.4gb], spins? [possibly], types [overlay]
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+mariajessie) starting as process 1 ...
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:28.334634362Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-07T15:33:46,128][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] heap size [1.9gb], compressed ordinary object pointers [true]
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:35.813448415Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-07T15:33:46,130][INFO ][o.e.n.Node ] node name [W3u251Y] derived from node ID [W3u251YpTuK25GAuM7O-ng]; set [node.name] to override
ui_1_c7e00a815717 | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Uses event mutexes
es_1_c00afa428f94 | [2018-12-07T15:33:46,130][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:43.004810852Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
ui_1_c7e00a815717 | 'Supervisord is running as root and it is searching '
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Compressed tables use zlib 1.2.8
es_1_c00afa428f94 | [2018-12-07T15:33:46,130][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:50.309950866Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
ui_1_c7e00a815717 | 2018-12-07 15:33:44,258 CRIT Supervisor running as root (no user in config file)
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Using Linux native AIO
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [aggs-matrix-stats]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:34:57.562354239Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
ui_1_c7e00a815717 | 2018-12-07 15:33:44,260 INFO supervisord started with pid 1
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Number of pools: 1
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [ingest-common]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:35:04.956597664Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
ui_1_c7e00a815717 | 2018-12-07 15:33:45,262 INFO spawned: 'webui' with pid 9
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Using SSE2 crc32 instructions
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-expression]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:35:12.103780063Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
ui_1_c7e00a815717 | 2018-12-07 15:33:46,470 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-groovy]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:35:19.262813137Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Completed initialization of buffer pool
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:35:26.72686607Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-mustache]
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139770535020288 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-07T15:35:34.252869193Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-painless]
sql_1_baad39f9a608 | 2018-12-07 15:33:44 139771256358784 [Note] InnoDB: Highest supported file format is Barracuda.
api_1_d0ea38fddddd | {"level":"error","time":"2018-12-07T15:35:36.25299783Z","message":"Failed to connect to ElasticSearch database. Not retrying."}
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [parent-join]
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] InnoDB: 128 out of 128 rollback segments are active.
api_1_d0ea38fddddd | panic: runtime error: invalid memory address or nil pointer dereference
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [percolator]
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] InnoDB: Creating shared tablespace for temporary tables
api_1_d0ea38fddddd | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a0f33]
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [reindex]
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
api_1_d0ea38fddddd |
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty3]
api_1_d0ea38fddddd | goroutine 1 [running]:
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] InnoDB: 5.7.19 started; log sequence number 2179744
es_1_c00afa428f94 | [2018-12-07T15:33:47,293][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty4]
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc4204feea0, 0x108a829, 0x3, 0xc42046de40, 0x14, 0xc420554b70, 0xe06aa0, 0xc420323cc0, ...)
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139770156009216 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
es_1_c00afa428f94 | [2018-12-07T15:33:47,294][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-geoip]
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] Plugin 'FEEDBACK' is disabled.
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequest(0x0, 0x189d380, 0xc4204feea0, 0x108a829, 0x3, 0xc42046de40, 0x14, 0xc420554b70, 0xe06aa0, 0xc420323cc0, ...)
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] Server socket created on IP: '::'.
es_1_c00afa428f94 | [2018-12-07T15:33:47,294][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-user-agent]
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Warning] 'proxies_priv' entry '@% root@397e1235ec88' ignored in --skip-name-resolve mode.
es_1_c00afa428f94 | [2018-12-07T15:33:47,294][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [x-pack]
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc420599e38, 0x189d380, 0xc4204feea0, 0x189d380, 0xc4204feea0, 0xc420323c80)
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] Reading of all Master_info entries succeded
es_1_c00afa428f94 | [2018-12-07T15:33:49,772][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/57] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] Added new Master_info '' to hash table
es_1_c00afa428f94 | [2018-12-07T15:33:49,799][INFO ][o.e.d.DiscoveryModule ] [W3u251Y] using discovery type [zen]
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1()
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139771256358784 [Note] mysqld: ready for connections.
es_1_c00afa428f94 | [2018-12-07T15:33:50,431][INFO ][o.e.n.Node ] initialized
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea
sql_1_baad39f9a608 | Version: '10.2.9-MariaDB-10.2.9+mariajessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
es_1_c00afa428f94 | [2018-12-07T15:33:50,431][INFO ][o.e.n.Node ] [W3u251Y] starting ...
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init()
sql_1_baad39f9a608 | 2018-12-07 15:33:45 139770156009216 [Note] InnoDB: Buffer pool(s) load completed at 181207 15:33:45
es_1_c00afa428f94 | [2018-12-07T15:33:50,627][INFO ][o.e.t.TransportService ] [W3u251Y] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300}
api_1_d0ea38fddddd | :1 +0x183
sql_1_baad39f9a608 | 2018-12-07 15:35:36 139770621622016 [Warning] Aborted connection 8 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets)
es_1_c00afa428f94 | [2018-12-07T15:33:50,635][INFO ][o.e.b.BootstrapChecks ] [W3u251Y] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init()
sql_1_baad39f9a608 | 2018-12-10 9:02:39 139770621622016 [Warning] Aborted connection 9 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets)
es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed
api_1_d0ea38fddddd | :1 +0x8c
es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
api_1_d0ea38fddddd | main.init()
es_1_c00afa428f94 | [2018-12-07T15:33:50,643][INFO ][o.e.n.Node ] [W3u251Y] stopping ...
api_1_d0ea38fddddd | :1 +0x78
es_1_c00afa428f94 | [2018-12-07T15:33:50,736][INFO ][o.e.n.Node ] [W3u251Y] stopped
api_1_d0ea38fddddd | {"level":"info","time":"2018-12-10T09:00:48.819069531Z","message":"Successfully connected to SQL database."}
es_1_c00afa428f94 | [2018-12-07T15:33:50,736][INFO ][o.e.n.Node ] [W3u251Y] closing ...
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:00:53.87354117Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-07T15:33:50,745][INFO ][o.e.n.Node ] [W3u251Y] closed
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:01.282592249Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-07T15:33:50,746][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:08.929072086Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:50,591][INFO ][o.e.n.Node ] [] initializing ...
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:16.365412968Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:23.89397026Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:50,651][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] using [1] data paths, mounts [[/ (overlay)]], net usable_space [5gb], net total_space [29.4gb], spins? [possibly], types [overlay]
es_1_c00afa428f94 | [2018-12-10T09:00:50,651][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] heap size [1.9gb], compressed ordinary object pointers [true]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:31.411812725Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:50,652][INFO ][o.e.n.Node ] node name [W3u251Y] derived from node ID [W3u251YpTuK25GAuM7O-ng]; set [node.name] to override
es_1_c00afa428f94 | [2018-12-10T09:00:50,652][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:38.941477556Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:50,652][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:46.316465352Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:51,810][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [aggs-matrix-stats]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:01:53.781694708Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:51,810][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [ingest-common]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:01.259192326Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:51,810][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-expression]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:08.70139361Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:51,810][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-groovy]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:16.005113075Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:51,810][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-mustache]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:23.234197869Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-painless]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:30.39815982Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [parent-join]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:02:37.897757971Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [percolator]
api_1_d0ea38fddddd | {"level":"error","time":"2018-12-10T09:02:39.897985891Z","message":"Failed to connect to ElasticSearch database. Not retrying."}
es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [reindex]
api_1_d0ea38fddddd | panic: runtime error: invalid memory address or nil pointer dereference
es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty3]
api_1_d0ea38fddddd | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a0f33]
es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty4]
api_1_d0ea38fddddd |
es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-geoip]
es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-user-agent]
api_1_d0ea38fddddd | goroutine 1 [running]:
es_1_c00afa428f94 | [2018-12-10T09:00:51,811][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [x-pack]
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc42055b800, 0x108a829, 0x3, 0xc4202fc440, 0x14, 0xc4204dc3c0, 0xe06aa0, 0xc42046c160, ...)
es_1_c00afa428f94 | [2018-12-10T09:00:54,314][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/55] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequest(0x0, 0x189d380, 0xc42055b800, 0x108a829, 0x3, 0xc4202fc440, 0x14, 0xc4204dc3c0, 0xe06aa0, 0xc42046c160, ...)
es_1_c00afa428f94 | [2018-12-10T09:00:54,348][INFO ][o.e.d.DiscoveryModule ] [W3u251Y] using discovery type [zen]
es_1_c00afa428f94 | [2018-12-10T09:00:54,969][INFO ][o.e.n.Node ] initialized
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f
es_1_c00afa428f94 | [2018-12-10T09:00:54,969][INFO ][o.e.n.Node ] [W3u251Y] starting ...
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc420599e38, 0x189d380, 0xc42055b800, 0x189d380, 0xc42055b800, 0xc42046c100)
es_1_c00afa428f94 | [2018-12-10T09:00:55,164][INFO ][o.e.t.TransportService ] [W3u251Y] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300}
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111
es_1_c00afa428f94 | [2018-12-10T09:00:55,170][INFO ][o.e.b.BootstrapChecks ] [W3u251Y] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1()
es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea
es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init()
es_1_c00afa428f94 | [2018-12-10T09:00:55,177][INFO ][o.e.n.Node ] [W3u251Y] stopping ...
api_1_d0ea38fddddd | :1 +0x183
es_1_c00afa428f94 | [2018-12-10T09:00:55,253][INFO ][o.e.n.Node ] [W3u251Y] stopped
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init()
es_1_c00afa428f94 | [2018-12-10T09:00:55,253][INFO ][o.e.n.Node ] [W3u251Y] closing ...
api_1_d0ea38fddddd | :1 +0x8c
es_1_c00afa428f94 | [2018-12-10T09:00:55,260][INFO ][o.e.n.Node ] [W3u251Y] closed
api_1_d0ea38fddddd | main.init()
api_1_d0ea38fddddd | :1 +0x78
es_1_c00afa428f94 | [2018-12-10T09:00:55,262][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
api_1_d0ea38fddddd | {"level":"info","time":"2018-12-10T09:03:19.337682382Z","message":"Successfully connected to SQL database."}
es_1_c00afa428f94 | [2018-12-10T09:03:21,082][INFO ][o.e.n.Node ] [] initializing ...
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:03:24.393296692Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:03:21,135][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] using [1] data paths, mounts [[/ (overlay)]], net usable_space [4.6gb], net total_space [29.4gb], spins? [possibly], types [overlay]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:03:31.756960454Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:03:21,135][INFO ][o.e.e.NodeEnvironment ] [W3u251Y] heap size [1.9gb], compressed ordinary object pointers [true]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:03:39.032258799Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:03:21,136][INFO ][o.e.n.Node ] node name [W3u251Y] derived from node ID [W3u251YpTuK25GAuM7O-ng]; set [node.name] to override
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:03:46.260197022Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:03:53.553834862Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:03:21,136][INFO ][o.e.n.Node ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.72-68.55.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:04:00.791731735Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1_c00afa428f94 | [2018-12-10T09:03:21,136][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [aggs-matrix-stats]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [ingest-common]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-expression]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-groovy]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-mustache]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [lang-painless]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [parent-join]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [percolator]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [reindex]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty3]
es_1_c00afa428f94 | [2018-12-10T09:03:22,156][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded module [transport-netty4]
es_1_c00afa428f94 | [2018-12-10T09:03:22,157][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-geoip]
es_1_c00afa428f94 | [2018-12-10T09:03:22,157][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [ingest-user-agent]
es_1_c00afa428f94 | [2018-12-10T09:03:22,157][INFO ][o.e.p.PluginsService ] [W3u251Y] loaded plugin [x-pack]
es_1_c00afa428f94 | [2018-12-10T09:03:24,671][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/56] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
es_1_c00afa428f94 | [2018-12-10T09:03:24,700][INFO ][o.e.d.DiscoveryModule ] [W3u251Y] using discovery type [zen]
es_1_c00afa428f94 | [2018-12-10T09:03:25,200][INFO ][o.e.n.Node ] initialized
es_1_c00afa428f94 | [2018-12-10T09:03:25,200][INFO ][o.e.n.Node ] [W3u251Y] starting ...
es_1_c00afa428f94 | [2018-12-10T09:03:25,349][INFO ][o.e.t.TransportService ] [W3u251Y] publish_address {10.19.84.4:9300}, bound_addresses {0.0.0.0:9300}
es_1_c00afa428f94 | [2018-12-10T09:03:25,355][INFO ][o.e.b.BootstrapChecks ] [W3u251Y] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed
es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
es_1_c00afa428f94 | [2018-12-10T09:03:25,363][INFO ][o.e.n.Node ] [W3u251Y] stopping ...
es_1_c00afa428f94 | [2018-12-10T09:03:25,463][INFO ][o.e.n.Node ] [W3u251Y] stopped
es_1_c00afa428f94 | [2018-12-10T09:03:25,463][INFO ][o.e.n.Node ] [W3u251Y] closing ...
es_1_c00afa428f94 | [2018-12-10T09:03:25,470][INFO ][o.e.n.Node ] [W3u251Y] closed
es_1_c00afa428f94 | [2018-12-10T09:03:25,472][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
trackit_es_1_c00afa428f94 exited with code 78
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:04:08.23391975Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:04:15.423108323Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}

I am using amazon linux to test this.

@sathyajackson
Copy link
Author

api_1_d0ea38fddddd | goroutine 1 [running]:
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequestWithOptions(0x0, 0x189d380, 0xc4205a0e40, 0x108a829, 0x3, 0xc420528ac0, 0x14, 0xc42059def0, 0xe06aa0, 0xc420437730, ...)
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequest(0x0, 0x189d380, 0xc4205a0e40, 0x108a829, 0x3, 0xc420528ac0, 0x14, 0xc42059def0, 0xe06aa0, 0xc420437730, ...)
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f
api_1_d0ea38fddddd | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc42054be38, 0x189d380, 0xc4205a0e40, 0x189d380, 0xc4205a0e40, 0xc4204376f0)
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init.1()
api_1_d0ea38fddddd | /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/s3.init()
api_1_d0ea38fddddd | :1 +0x183
api_1_d0ea38fddddd | github.com/trackit/trackit-server/aws/routes.init()
api_1_d0ea38fddddd | :1 +0x8c
api_1_d0ea38fddddd | main.init()
api_1_d0ea38fddddd | :1 +0x78
sql_1_baad39f9a608 | 2018-12-10 9:05:08 139770621622016 [Warning] Aborted connection 10 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets)

@sathyajackson
Copy link
Author

docker ps output

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ef3d4e53cd3b trackit_sql "docker-entrypoint.s…" 2 days ago Up 2 days 127.0.0.1:3306->3306/tcp trackit_sql_1_baad39f9a608
8aaf42dd85ce trackit_ui "/usr/bin/supervisord" 2 days ago Up 2 days 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp trackit_ui_1_c7e00a815717

@sathyajackson
Copy link
Author

I did make some modification on the docker-compose file to point the API to server public IP. Still i see the below error and my ealstic container is not running.

api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:07:23.123310772Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.4:9200: getsockopt: connection refused: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:07:30.404844304Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:07:37.880940548Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:07:45.299431229Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:07:52.662917352Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:00.176747145Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:07.420307808Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:14.782587565Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:22.05947155Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:29.313760388Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:36.713434361Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:43.874558415Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:51.17893862Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:08:58.227660307Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}

@sathyajackson
Copy link
Author

Not sure if this is related to the problem. but here is the output i get when i try everything fresh.

es_1_c00afa428f94 | ERROR: [1] bootstrap checks failed
es_1_c00afa428f94 | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
es_1_c00afa428f94 | [2018-12-10T09:40:57,584][INFO ][o.e.n.Node ] [W3u251Y] stopping ...
es_1_c00afa428f94 | [2018-12-10T09:40:57,684][INFO ][o.e.n.Node ] [W3u251Y] stopped
es_1_c00afa428f94 | [2018-12-10T09:40:57,684][INFO ][o.e.n.Node ] [W3u251Y] closing ...
es_1_c00afa428f94 | [2018-12-10T09:40:57,692][INFO ][o.e.n.Node ] [W3u251Y] closed
es_1_c00afa428f94 | [2018-12-10T09:40:57,694][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
trackit_es_1_c00afa428f94 exited with code 78
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:42:16.356009835Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:42:23.416671448Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
api_1_d0ea38fddddd | {"level":"warning","time":"2018-12-10T09:42:30.583070176Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}

@sathyajackson
Copy link
Author

i tried to install the same in a fresh machine and i am now getting the below python execption.

===> pulling the docker containers
Traceback (most recent call last):
File "./trackit-server/scripts/awsenv", line 31, in
main(profile, cmd)
File "./trackit-server/scripts/awsenv", line 16, in main
profile_config = get_profile_config(profile)
File "./trackit-server/scripts/awsenv", line 13, in get_profile_config
return c[profile]
File "/usr/lib64/python3.6/configparser.py", line 959, in getitem
raise KeyError(key)
KeyError: 'default'
===> building containers
Traceback (most recent call last):
File "./trackit-server/scripts/awsenv", line 31, in
main(profile, cmd)
File "./trackit-server/scripts/awsenv", line 16, in main
profile_config = get_profile_config(profile)
File "./trackit-server/scripts/awsenv", line 13, in get_profile_config
return c[profile]
File "/usr/lib64/python3.6/configparser.py", line 959, in getitem
raise KeyError(key)
KeyError: 'default'
===> starting trackit
Traceback (most recent call last):
File "./trackit-server/scripts/awsenv", line 31, in
main(profile, cmd)
File "./trackit-server/scripts/awsenv", line 16, in main
profile_config = get_profile_config(profile)
File "./trackit-server/scripts/awsenv", line 13, in get_profile_config
return c[profile]
File "/usr/lib64/python3.6/configparser.py", line 959, in getitem
raise KeyError(key)
KeyError: 'default'

@thibautcornolti
Copy link
Member

thibautcornolti commented Dec 20, 2018 •
edited
Loading

You need to configure aws-cli: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
Then, try again sudo ulimit -n 65536 and sudo sysctl -w fs.file-max=65536

Please write your logs in the appropriate format so we can read it.

image
will be shown as:

[my logs]

@sathyajackson
Copy link
Author

Hi thanks for the input. I have already configured the Access_key and secreat_key using aws cli on the same server where i am running trackit.

Still when i start the docker containers i am getting the below message

ARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.

@thibautcornolti
Copy link
Member

Hi, are you sure you start TrackIt with our start.sh script?

@sathyajackson
Copy link
Author

Yes I do with start.sh

@thibautcornolti
Copy link
Member

Look in the file ~/.aws/credentials, do you have a default profile?

@sathyajackson
Copy link
Author

Yes, I do have that and I even check the credentials and I works fine with aws-cli

@thibautcornolti
Copy link
Member

./trackit-server/scripts/awsenv default env | grep 'AWS_' | cut -f1 -d"="

Does it display this?

AWS_DEFAULT_REGION
AWS_ACCESS_KEY_ID
AWS_SESSION_TOKEN
AWS_SECRET_ACCESS_KEY

@sathyajackson
Copy link
Author

I get the following error

Traceback (most recent call last):
File "./awsenv", line 31, in
main(profile, cmd)
File "./awsenv", line 16, in main
profile_config = get_profile_config(profile)
File "./awsenv", line 13, in get_profile_config
return c[profile]
File "/usr/lib64/python3.6/configparser.py", line 959, in getitem
raise KeyError(key)
KeyError: 'default'

@sathyajackson
Copy link
Author

sorry for the above statement.

Now i got to see what is expected.

AWS_CLOUDWATCH_HOME
AWS_PATH
AWS_AUTO_SCALING_HOME
AWS_ELB_HOME
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

@thibautcornolti
Copy link
Member

thibautcornolti commented Jan 3, 2019 •
edited
Loading

Ok!

So let's start again, what are the start.sh and the containers outputs?
Please write your logs as I wrote here trackit/trackit#69 (comment)

Thanks

@sathyajackson
Copy link
Author

sathyajackson commented Jan 3, 2019 •
edited by thibautcornolti
Loading

Alrite here you go.

./start.sh
===> checking vm.max_map_count setting
===> updating git submodules
Adding existing repo at 'trackit2-client' to the index
Adding existing repo at 'trackit-server' to the index
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), done.
From https://github.com/trackit/trackit-server
   3b379b6..e1b2aca  master     -> origin/master
Submodule path 'trackit-server': checked out 'e1b2acafc2d678932d0b97d46aa3c6debe9b4258'
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), done.
From https://github.com/trackit/trackit2-client
   5f433fb..a3b72bc  master     -> origin/master
Submodule path 'trackit2-client': checked out 'a3b72bc0a1a83824a2ce923b4907618fb16ac068'
/data/trackit/scripts /data/trackit
===> building intermediate container for server compilation
Sending build context to Docker daemon  9.185MB
Step 1/13 : FROM ubuntu:16.04
 ---> a51debf7e1eb
Step 2/13 : MAINTAINER Romain Zanchi <[email protected]>
 ---> Using cache
 ---> e63c6835f86f
Step 3/13 : RUN apt-get update
 ---> Using cache
 ---> aec3f4a32c4f
Step 4/13 : RUN apt-get install -y curl git
 ---> Using cache
 ---> c1147dd473cb
Step 5/13 : WORKDIR /
 ---> Using cache
 ---> 3df8f0292676
Step 6/13 : RUN curl -O https://dl.google.com/go/go1.9.2.linux-amd64.tar.gz
 ---> Using cache
 ---> 93a76e81b7f9
Step 7/13 : RUN tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz
 ---> Using cache
 ---> ae722739f0e1
Step 8/13 : ENV PATH="/usr/local/go/bin:${PATH}"
 ---> Using cache
 ---> 13ca92816530
Step 9/13 : RUN go get -u github.com/kardianos/govendor
 ---> Using cache
 ---> 44a831cbcbcb
Step 10/13 : ENV PATH="/root/go/bin:${PATH}"
 ---> Using cache
 ---> 7ab7a1ad79a1
Step 11/13 : COPY . /root/go/src/github.com/trackit
 ---> 4e45d507d712
Step 12/13 : COPY compiler/compile_server.sh /compile_server.sh
 ---> 96420754d1be
Step 13/13 : ENTRYPOINT ["/compile_server.sh"]
 ---> Running in 639b029ff2f9
Removing intermediate container 639b029ff2f9
 ---> 8ac7cd80b58b
Successfully built 8ac7cd80b58b
Successfully tagged server_compiler:latest
===> fetching dependencies
fetch "github.com/alexflint/go-arg"
fetch "github.com/alexflint/go-scalar"
fetch "github.com/aws/aws-sdk-go/aws"
fetch "github.com/aws/aws-sdk-go/aws/awserr"
fetch "github.com/aws/aws-sdk-go/aws/awsutil"
fetch "github.com/aws/aws-sdk-go/aws/client"
fetch "github.com/aws/aws-sdk-go/aws/client/metadata"
fetch "github.com/aws/aws-sdk-go/aws/corehandlers"
fetch "github.com/aws/aws-sdk-go/aws/credentials"
fetch "github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
fetch "github.com/aws/aws-sdk-go/aws/credentials/endpointcreds"
fetch "github.com/aws/aws-sdk-go/aws/credentials/stscreds"
fetch "github.com/aws/aws-sdk-go/aws/csm"
fetch "github.com/aws/aws-sdk-go/aws/defaults"
fetch "github.com/aws/aws-sdk-go/aws/ec2metadata"
fetch "github.com/aws/aws-sdk-go/aws/endpoints"
fetch "github.com/aws/aws-sdk-go/aws/request"
fetch "github.com/aws/aws-sdk-go/aws/session"
fetch "github.com/aws/aws-sdk-go/aws/signer/v4"
fetch "github.com/aws/aws-sdk-go/internal/sdkio"
fetch "github.com/aws/aws-sdk-go/internal/sdkrand"
fetch "github.com/aws/aws-sdk-go/internal/sdkuri"
fetch "github.com/aws/aws-sdk-go/internal/shareddefaults"
fetch "github.com/aws/aws-sdk-go/private/protocol"
fetch "github.com/aws/aws-sdk-go/private/protocol/ec2query"
fetch "github.com/aws/aws-sdk-go/private/protocol/eventstream"
fetch "github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi"
fetch "github.com/aws/aws-sdk-go/private/protocol/json/jsonutil"
fetch "github.com/aws/aws-sdk-go/private/protocol/json/jsonutil"
fetch "github.com/aws/aws-sdk-go/private/protocol/jsonrpc"
fetch "github.com/aws/aws-sdk-go/private/protocol/jsonrpc"
fetch "github.com/aws/aws-sdk-go/private/protocol/query"
fetch "github.com/aws/aws-sdk-go/private/protocol/query/queryutil"
fetch "github.com/aws/aws-sdk-go/private/protocol/rest"
fetch "github.com/aws/aws-sdk-go/private/protocol/restjson"
fetch "github.com/aws/aws-sdk-go/private/protocol/restxml"
fetch "github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil"
fetch "github.com/aws/aws-sdk-go/service/cloudwatch"
fetch "github.com/aws/aws-sdk-go/service/costexplorer"
fetch "github.com/aws/aws-sdk-go/service/ec2"
fetch "github.com/aws/aws-sdk-go/service/elasticache"
fetch "github.com/aws/aws-sdk-go/service/elasticsearchservice"
fetch "github.com/aws/aws-sdk-go/service/lambda"
fetch "github.com/aws/aws-sdk-go/service/marketplaceentitlementservice"
fetch "github.com/aws/aws-sdk-go/service/marketplacemetering"
fetch "github.com/aws/aws-sdk-go/service/marketplacemetering"
fetch "github.com/aws/aws-sdk-go/service/organizations"
fetch "github.com/aws/aws-sdk-go/service/rds"
fetch "github.com/aws/aws-sdk-go/service/s3"
fetch "github.com/aws/aws-sdk-go/service/s3/s3iface"
fetch "github.com/aws/aws-sdk-go/service/s3/s3manager"
fetch "github.com/aws/aws-sdk-go/service/sts"
fetch "github.com/denisenkom/go-mssqldb"
fetch "github.com/dgrijalva/jwt-go"
fetch "github.com/gedex/inflector"
fetch "github.com/go-ini/ini"
fetch "github.com/go-sql-driver/mysql"
fetch "github.com/jmespath/go-jmespath"
fetch "github.com/knq/dburl"
fetch "github.com/knq/snaker"
fetch "github.com/knq/xo"
fetch "github.com/knq/xo/internal"
fetch "github.com/knq/xo/loaders"
fetch "github.com/knq/xo/models"
fetch "github.com/knq/xo/tplbin"
fetch "github.com/knq/xoutil"
fetch "github.com/lib/pq"
fetch "github.com/lib/pq/oid"
fetch "github.com/mailru/easyjson"
fetch "github.com/mailru/easyjson/buffer"
fetch "github.com/mailru/easyjson/jlexer"
fetch "github.com/mailru/easyjson/jwriter"
fetch "github.com/mattn/go-sqlite3"
fetch "github.com/pkg/errors"
fetch "github.com/satori/go.uuid"
fetch "github.com/sha1sum/aws_signing_client"
fetch "github.com/tealeg/xlsx"
fetch "github.com/trackit/jsonlog"
fetch "golang.org/x/crypto/bcrypt"
fetch "golang.org/x/crypto/blowfish"
fetch "golang.org/x/crypto/md4"
fetch "golang.org/x/net/context"
fetch "golang.org/x/sync/errgroup"
fetch "golang.org/x/tools/cmd/goimports"
fetch "golang.org/x/tools/go/ast/astutil"
fetch "golang.org/x/tools/imports"
fetch "gopkg.in/olivere/elastic.v5"
fetch "gopkg.in/olivere/elastic.v5/config"
fetch "gopkg.in/olivere/elastic.v5/uritemplates"
fetch "gopkg.in/rana/ora.v4"
fetch "gopkg.in/rana/ora.v4/date"
fetch "gopkg.in/rana/ora.v4/num"
===> compiling trackit server
+ rm -f ./server/main
+ [[ ! -f server/main ]]
+ pushd server
+ ./buildstatic.sh
~/go/src/github.com/trackit/trackit-server/server ~/go/src/github.com/trackit/trackit-server
++ git rev-parse --short=8 HEAD
+ build_number=e1b2acaf
++ git status --porcelain
++ wc -l
+ [[ 0 -gt 0 ]]
+ CGO_ENABLED=0
+ GOOS=linux
+ go build -a -installsuffix cgo -ldflags '-X main.buildNumber=e1b2acaf' -o main
~/go/src/github.com/trackit/trackit-server
+ popd
+ mv ./server/main /trackit-server
/data/trackit
/data/trackit/trackit-server/scripts /data/trackit
==> copying schema.sql
+ cp ../db/schema.sql ../docker/sql/schema.sql
/data/trackit
===> pulling the docker containers
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
Pulling ui  ... done
Pulling es  ... done
Pulling sql ... done
Pulling api ... done
===> building containers
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
es uses an image, skipping
Building ui
Step 1/14 : FROM node:6-slim
 ---> 39cfb4866972
Step 2/14 : ARG API_URL
 ---> Using cache
 ---> d0881fb735be
Step 3/14 : ENV API_URL=${API_URL}
 ---> Using cache
 ---> 1d4053b7669d
Step 4/14 : RUN curl -sS http://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
 ---> Using cache
 ---> c35e9d70f1d9
Step 5/14 : RUN echo "deb http://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
 ---> Using cache
 ---> 00d09b07f70f
Step 6/14 : RUN apt-get update && apt-get install -y supervisor git
 ---> Using cache
 ---> c678367493e1
Step 7/14 : RUN mkdir -p /var/log/supervisor
 ---> Using cache
 ---> ad72386c56bf
Step 8/14 : COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
 ---> Using cache
 ---> 39c9ea2aa904
Step 9/14 : WORKDIR /webui
 ---> Using cache
 ---> 228bfd8f714f
Step 10/14 : COPY ./ ./
 ---> 8e461f2d3a14
Step 11/14 : RUN yarn install
 ---> Running in be333f95dd69
yarn install v1.12.3
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 167.28s.
Removing intermediate container be333f95dd69
 ---> 465aeb9be3de
Step 12/14 : RUN yarn run build
 ---> Running in 517da548a7af
yarn run v1.12.3
$ node scripts/build.js
Creating an optimized production build...
Compiled successfully.

File sizes after gzip:

  529.02 KB  build/static/js/main.b4beb5e8.js
  47.02 KB   build/static/css/main.7a79ac8c.css

The bundle size is significantly larger than recommended.
Consider reducing it with code splitting: https://goo.gl/9VhYWB
You can also analyze the project dependencies: https://goo.gl/LeUzfb

The project was built assuming it is hosted at /.
You can control this with the homepage field in your package.json.

The build folder is ready to be deployed.
You may serve it with a static server:

  yarn global add serve
  serve -s build

Done in 72.37s.
Removing intermediate container 517da548a7af
 ---> 778e9253d0f1
Step 13/14 : EXPOSE 80 443
 ---> Running in b1888e79231c
Removing intermediate container b1888e79231c
 ---> 516c39c06f46
Step 14/14 : CMD ["/usr/bin/supervisord"]
 ---> Running in e0b192e55350
Removing intermediate container e0b192e55350
 ---> 96ccf2bb4036
Successfully built 96ccf2bb4036
Successfully tagged trackit_ui:latest
Building sql
Step 1/3 : FROM mariadb:10.2.9
 ---> 629d7f88b09c
Step 2/3 : MAINTAINER Victor Schubert <[email protected]>
 ---> Using cache
 ---> 4884858e75c0
Step 3/3 : COPY schema.sql /docker-entrypoint-initdb.d/schema.sql
 ---> f6a1e98c1f2d
Successfully built f6a1e98c1f2d
Successfully tagged trackit_sql:latest
Building api
Step 1/6 : FROM scratch
 --->
Step 2/6 : MAINTAINER Victor Schubert <[email protected]>
 ---> Using cache
 ---> bcfaae81a786
Step 3/6 : EXPOSE 80
 ---> Using cache
 ---> 2a1d3558c4b5
Step 4/6 : COPY ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
 ---> Using cache
 ---> f38cf1c6e20b
Step 5/6 : COPY main /main
 ---> 60d2d487a1d1
Step 6/6 : ENTRYPOINT ["/main"]
 ---> Running in 07a47a834966
Removing intermediate container 07a47a834966
 ---> fe4a1bbd59f2
Successfully built fe4a1bbd59f2
Successfully tagged trackit_api:latest
===> starting trackit
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
Creating network "trackit_app" with driver "bridge"
Creating trackit_sql_1 ... done
Creating trackit_ui_1  ... done
Creating trackit_es_1  ... done
Creating trackit_api_1 ... done
===> installation completed, TrackIt is now running on port 80

@thibautcornolti
Copy link
Member

Ok! Everything seems to have started well.

Could you send me the containers logs now? Get them by typing docker-compose logs -f
Still can't login?

@sathyajackson
Copy link
Author

sathyajackson commented Jan 3, 2019 •
edited by thibautcornolti
Loading

Yes still cant login here are the logs of docker-compose.

WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
Attaching to trackit_api_1, trackit_ui_1, trackit_es_1, trackit_sql_1
sql_1  | Initializing database
sql_1  |
sql_1  | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
sql_1  | To do so, start the server, then issue the following commands:
api_1  | {"level":"warning","time":"2019-01-03T12:39:03.303926336Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
api_1  | {"level":"warning","time":"2019-01-03T12:39:05.305095525Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
api_1  | {"level":"warning","time":"2019-01-03T12:39:07.311573427Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
api_1  | {"level":"warning","time":"2019-01-03T12:39:09.312323275Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
sql_1  |
es_1   | [2019-01-03T12:39:05,384][INFO ][o.e.n.Node               ] [] initializing ...
ui_1   | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
ui_1   |   'Supervisord is running as root and it is searching '
sql_1  | '/usr/bin/mysqladmin' -u root password 'new-password'
es_1   | [2019-01-03T12:39:05,464][INFO ][o.e.e.NodeEnvironment    ] [SDWVkXe] using [1] data paths, mounts [[/ (overlay)]], net usable_space [5.7gb], net total_space [29.4gb], spins? [possibly], types [overlay]
ui_1   | 2019-01-03 12:39:02,601 CRIT Supervisor running as root (no user in config file)
api_1  | {"level":"warning","time":"2019-01-03T12:39:11.313068565Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
sql_1  | '/usr/bin/mysqladmin' -u root -h  password 'new-password'
sql_1  |
ui_1   | 2019-01-03 12:39:02,640 INFO supervisord started with pid 1
api_1  | {"level":"warning","time":"2019-01-03T12:39:13.313837089Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.2:3306: getsockopt: connection refused"}
sql_1  | Alternatively you can run:
es_1   | [2019-01-03T12:39:05,464][INFO ][o.e.e.NodeEnvironment    ] [SDWVkXe] heap size [1.9gb], compressed ordinary object pointers [true]
ui_1   | 2019-01-03 12:39:03,642 INFO spawned: 'webui' with pid 8
ui_1   | 2019-01-03 12:39:05,481 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
sql_1  | '/usr/bin/mysql_secure_installation'
es_1   | [2019-01-03T12:39:05,465][INFO ][o.e.n.Node               ] node name [SDWVkXe] derived from node ID [SDWVkXeIS7O3bPvhaSB7jA]; set [node.name] to override
api_1  | {"level":"info","time":"2019-01-03T12:39:15.31513217Z","message":"Successfully connected to SQL database."}
sql_1  |
es_1   | [2019-01-03T12:39:05,466][INFO ][o.e.n.Node               ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.77-70.59.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
api_1  | {"level":"warning","time":"2019-01-03T12:39:20.939298355Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | which will also give you the option of removing the test
es_1   | [2019-01-03T12:39:05,466][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
api_1  | {"level":"warning","time":"2019-01-03T12:39:28.44700456Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | databases and anonymous user created by default.  This is
es_1   | [2019-01-03T12:39:10,562][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [aggs-matrix-stats]
api_1  | {"level":"warning","time":"2019-01-03T12:39:36.078019018Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | strongly recommended for production servers.
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [ingest-common]
api_1  | {"level":"warning","time":"2019-01-03T12:39:43.636509336Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  |
sql_1  | See the MariaDB Knowledgebase at http://mariadb.com/kb or the
api_1  | {"level":"warning","time":"2019-01-03T12:39:51.009755884Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [lang-expression]
sql_1  | MySQL manual for more instructions.
api_1  | {"level":"warning","time":"2019-01-03T12:39:58.613938228Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [lang-groovy]
sql_1  |
api_1  | {"level":"warning","time":"2019-01-03T12:40:06.253430883Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [lang-mustache]
sql_1  | Please report any problems at http://mariadb.org/jira
api_1  | {"level":"warning","time":"2019-01-03T12:40:13.941900517Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [lang-painless]
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [parent-join]
api_1  | {"level":"warning","time":"2019-01-03T12:40:21.376197802Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  |
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [percolator]
api_1  | {"level":"warning","time":"2019-01-03T12:40:28.759457554Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | The latest information about MariaDB is available at http://mariadb.org/.
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [reindex]
api_1  | {"level":"warning","time":"2019-01-03T12:40:36.017083133Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | You can find additional information about the MySQL part at:
es_1   | [2019-01-03T12:39:10,563][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [transport-netty3]
api_1  | {"level":"warning","time":"2019-01-03T12:40:43.444806839Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | http://dev.mysql.com
es_1   | [2019-01-03T12:39:10,564][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded module [transport-netty4]
api_1  | {"level":"warning","time":"2019-01-03T12:40:50.874534064Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | Consider joining MariaDB's strong and vibrant community:
es_1   | [2019-01-03T12:39:10,564][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded plugin [ingest-geoip]
api_1  | {"level":"warning","time":"2019-01-03T12:40:58.342928551Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  | https://mariadb.org/get-involved/
es_1   | [2019-01-03T12:39:10,564][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded plugin [ingest-user-agent]
es_1   | [2019-01-03T12:39:10,564][INFO ][o.e.p.PluginsService     ] [SDWVkXe] loaded plugin [x-pack]
api_1  | {"level":"warning","time":"2019-01-03T12:41:05.818648543Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp: lookup es on 127.0.0.11:53: no such host: no Elasticsearch node available"}
sql_1  |
api_1  | {"level":"error","time":"2019-01-03T12:41:07.818852452Z","message":"Failed to connect to ElasticSearch database. Not retrying."}
sql_1  | Database initialized
es_1   | [2019-01-03T12:39:14,244][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/48] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
api_1  | panic: runtime error: invalid memory address or nil pointer dereference
sql_1  | MySQL init process in progress...
es_1   | [2019-01-03T12:39:14,291][INFO ][o.e.d.DiscoveryModule    ] [SDWVkXe] using discovery type [zen]
api_1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8a4d33]
api_1  |
es_1   | [2019-01-03T12:39:15,160][INFO ][o.e.n.Node               ] initialized
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 97 ...
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
es_1   | [2019-01-03T12:39:15,160][INFO ][o.e.n.Node               ] [SDWVkXe] starting ...
api_1  | goroutine 1 [running]:
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Uses event mutexes
es_1   | [2019-01-03T12:39:15,446][INFO ][o.e.t.TransportService   ] [SDWVkXe] publish_address {10.19.84.3:9300}, bound_addresses {0.0.0.0:9300}
api_1  | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequestWithOptions(0x0, 0x1a57380, 0xc42051a720, 0x119070c, 0x3, 0xc4203a6a40, 0x14, 0xc42026a0c0, 0xecebc0, 0xc4202cf1b0, ...)
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Compressed tables use zlib 1.2.8
es_1   | [2019-01-03T12:39:15,455][INFO ][o.e.b.BootstrapChecks    ] [SDWVkXe] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
api_1  |        /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1222 +0xe3
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Using Linux native AIO
es_1   | ERROR: [1] bootstrap checks failed
api_1  | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).PerformRequest(0x0, 0x1a57380, 0xc42051a720, 0x119070c, 0x3, 0xc4203a6a40, 0x14, 0xc42026a0c0, 0xecebc0, 0xc4202cf1b0, ...)
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Number of pools: 1
es_1   | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
api_1  |        /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/client.go:1190 +0x13f
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Using SSE2 crc32 instructions
es_1   | [2019-01-03T12:39:15,464][INFO ][o.e.n.Node               ] [SDWVkXe] stopping ...
api_1  | github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic%2ev5.(*IndicesPutTemplateService).Do(0xc4204d9e38, 0x1a57380, 0xc42051a720, 0x1a57380, 0xc42051a720, 0xc4202cf150)
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
es_1   | [2019-01-03T12:39:15,568][INFO ][o.e.n.Node               ] [SDWVkXe] stopped
api_1  |        /root/go/src/github.com/trackit/trackit-server/vendor/gopkg.in/olivere/elastic.v5/indices_put_template.go:184 +0x111
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Completed initialization of buffer pool
es_1   | [2019-01-03T12:39:15,568][INFO ][o.e.n.Node               ] [SDWVkXe] closing ...
api_1  | github.com/trackit/trackit-server/aws/s3.init.1()
es_1   | [2019-01-03T12:39:15,581][INFO ][o.e.n.Node               ] [SDWVkXe] closed
sql_1  | 2019-01-03 12:39:05 140581952399104 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
es_1   | [2019-01-03T12:39:15,583][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
api_1  |        /root/go/src/github.com/trackit/trackit-server/aws/s3/mappings.go:33 +0xea
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Highest supported file format is Barracuda.
api_1  | github.com/trackit/trackit-server/aws/s3.init()
api_1  |        <autogenerated>:1 +0x183
api_1  | github.com/trackit/trackit-server/aws/routes.init()
api_1  |        <autogenerated>:1 +0x8c
api_1  | main.init()
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: 128 out of 128 rollback segments are active.
trackit_es_1 exited with code 78
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Creating shared tablespace for temporary tables
api_1  |        <autogenerated>:1 +0x78
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] InnoDB: 5.7.19 started; log sequence number 1619987
sql_1  | 2019-01-03 12:39:05 140581711894272 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] Plugin 'FEEDBACK' is disabled.
sql_1  | 2019-01-03 12:39:05 140581711894272 [Note] InnoDB: Buffer pool(s) load completed at 190103 12:39:05
sql_1  | 2019-01-03 12:39:05 140582673598336 [Warning] 'user' entry 'root@6c70c7eb5849' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-03 12:39:05 140582673598336 [Warning] 'proxies_priv' entry '@% root@6c70c7eb5849' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] Reading of all Master_info entries succeded
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] Added new Master_info '' to hash table
sql_1  | 2019-01-03 12:39:05 140582673598336 [Note] mysqld: ready for connections.
sql_1  | Version: '10.2.9-MariaDB-10.2.9+maria~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
sql_1  | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
sql_1  | GENERATED ROOT PASSWORD: eefuthei9lohCuxohl9me0taegh0oini
sql_1  | 2019-01-03 12:39:09 140582537709312 [Warning] 'proxies_priv' entry '@% root@6c70c7eb5849' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-03 12:39:09 140582537709312 [Warning] 'proxies_priv' entry '@% root@6c70c7eb5849' ignored in --skip-name-resolve mode.
sql_1  |
sql_1  | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/schema.sql
sql_1  |
sql_1  |
sql_1  | 2019-01-03 12:39:11 140582537406208 [Note] mysqld (unknown): Normal shutdown
sql_1  |
sql_1  | 2019-01-03 12:39:11 140582537406208 [Note] Event Scheduler: Purging the queue. 0 events
sql_1  | 2019-01-03 12:39:11 140581829326592 [Note] InnoDB: FTS optimize thread exiting.
sql_1  | 2019-01-03 12:39:11 140582537406208 [Note] InnoDB: Starting shutdown...
sql_1  | 2019-01-03 12:39:11 140581711894272 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
sql_1  | 2019-01-03 12:39:11 140581711894272 [Note] InnoDB: Buffer pool(s) dump completed at 190103 12:39:11
sql_1  | 2019-01-03 12:39:13 140582537406208 [Note] InnoDB: Shutdown completed; log sequence number 2322244
sql_1  | 2019-01-03 12:39:13 140582537406208 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
sql_1  | 2019-01-03 12:39:13 140582537406208 [Note] mysqld: Shutdown complete
sql_1  |
sql_1  |
sql_1  | MySQL init process done. Ready for start up.
sql_1  |
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 1 ...
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
trackit_api_1 exited with code 2
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Uses event mutexes
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Compressed tables use zlib 1.2.8
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Using Linux native AIO
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Number of pools: 1
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Using SSE2 crc32 instructions
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Completed initialization of buffer pool
sql_1  | 2019-01-03 12:39:13 140656581883648 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Highest supported file format is Barracuda.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: 128 out of 128 rollback segments are active.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Creating shared tablespace for temporary tables
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: Waiting for purge to start
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] InnoDB: 5.7.19 started; log sequence number 2322244
sql_1  | 2019-01-03 12:39:13 140656345343744 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] Plugin 'FEEDBACK' is disabled.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] Server socket created on IP: '::'.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Warning] 'proxies_priv' entry '@% root@6c70c7eb5849' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] Reading of all Master_info entries succeded
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] Added new Master_info '' to hash table
sql_1  | 2019-01-03 12:39:13 140657307412352 [Note] mysqld: ready for connections.
sql_1  | Version: '10.2.9-MariaDB-10.2.9+maria~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
sql_1  | 2019-01-03 12:39:13 140656345343744 [Note] InnoDB: Buffer pool(s) load completed at 190103 12:39:13
sql_1  | 2019-01-03 12:41:07 140657165969152 [Warning] Aborted connection 8 to db: 'trackit' user: 'trackit' host: '10.19.84.5' (Got an error reading communication packets)

@sathyajackson
Copy link
Author

Please ignore my previous comment, I see the elastic search is not running. Let me fix that and update you with the latest logs.

@sathyajackson
Copy link
Author

sathyajackson commented Jan 4, 2019 •
edited by thibautcornolti
Loading

Here you go complete logs from all containers.

docker-compose logs -f
WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
Attaching to trackit_api_1, trackit_sql_1, trackit_es_1, trackit_ui_1
es_1   | [2019-01-04T08:26:31,805][INFO ][o.e.n.Node               ] [] initializing ...
es_1   | [2019-01-04T08:26:31,886][INFO ][o.e.e.NodeEnvironment    ] [qxFs6OU] using [1] data paths, mounts [[/ (overlay)]], net usable_space [4.4gb], net total_space [29.4gb], spins? [possibly], types [overlay]
es_1   | [2019-01-04T08:26:31,887][INFO ][o.e.e.NodeEnvironment    ] [qxFs6OU] heap size [1.9gb], compressed ordinary object pointers [true]
es_1   | [2019-01-04T08:26:31,888][INFO ][o.e.n.Node               ] node name [qxFs6OU] derived from node ID [qxFs6OUqTVCOiQXlkExiig]; set [node.name] to override
es_1   | [2019-01-04T08:26:31,888][INFO ][o.e.n.Node               ] version[5.5.3], pid[1], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.77-70.59.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
es_1   | [2019-01-04T08:26:31,888][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [aggs-matrix-stats]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [ingest-common]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [lang-expression]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [lang-groovy]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [lang-mustache]
es_1   | [2019-01-04T08:26:36,661][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [lang-painless]
es_1   | [2019-01-04T08:26:36,662][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [parent-join]
es_1   | [2019-01-04T08:26:36,663][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [percolator]
es_1   | [2019-01-04T08:26:36,663][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [reindex]
es_1   | [2019-01-04T08:26:36,663][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [transport-netty3]
es_1   | [2019-01-04T08:26:36,663][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded module [transport-netty4]
es_1   | [2019-01-04T08:26:36,664][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded plugin [ingest-geoip]
es_1   | [2019-01-04T08:26:36,664][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded plugin [ingest-user-agent]
es_1   | [2019-01-04T08:26:36,664][INFO ][o.e.p.PluginsService     ] [qxFs6OU] loaded plugin [x-pack]
es_1   | [2019-01-04T08:26:40,463][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/48] [Main.cc@128] controller (64 bit): Version 5.5.3 (Build c18fe968cb6cd4) Copyright (c) 2017 Elasticsearch BV
es_1   | [2019-01-04T08:26:40,522][INFO ][o.e.d.DiscoveryModule    ] [qxFs6OU] using discovery type [zen]
es_1   | [2019-01-04T08:26:41,559][INFO ][o.e.n.Node               ] initialized
es_1   | [2019-01-04T08:26:41,559][INFO ][o.e.n.Node               ] [qxFs6OU] starting ...
ui_1   | /usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
ui_1   |   'Supervisord is running as root and it is searching '
es_1   | [2019-01-04T08:26:41,822][INFO ][o.e.t.TransportService   ] [qxFs6OU] publish_address {10.19.84.3:9300}, bound_addresses {0.0.0.0:9300}
api_1  | {"level":"warning","time":"2019-01-04T08:26:29.248415991Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.4:3306: getsockopt: connection refused"}
api_1  | {"level":"warning","time":"2019-01-04T08:26:31.249527361Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.4:3306: getsockopt: connection refused"}
es_1   | [2019-01-04T08:26:41,833][INFO ][o.e.b.BootstrapChecks    ] [qxFs6OU] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ui_1   | 2019-01-04 08:26:28,881 CRIT Supervisor running as root (no user in config file)
sql_1  | Initializing database
api_1  | {"level":"warning","time":"2019-01-04T08:26:33.250291638Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.4:3306: getsockopt: connection refused"}
es_1   | [2019-01-04T08:26:44,905][INFO ][o.e.c.s.ClusterService   ] [qxFs6OU] new_master {qxFs6OU}{qxFs6OUqTVCOiQXlkExiig}{Aj1med4IQKiLrobsQj7XRg}{10.19.84.3}{10.19.84.3:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
ui_1   | 2019-01-04 08:26:28,920 INFO supervisord started with pid 1
sql_1  |
api_1  | {"level":"warning","time":"2019-01-04T08:26:35.2510975Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.4:3306: getsockopt: connection refused"}
es_1   | [2019-01-04T08:26:44,965][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [qxFs6OU] publish_address {10.19.84.3:9200}, bound_addresses {0.0.0.0:9200}
ui_1   | 2019-01-04 08:26:29,926 INFO spawned: 'webui' with pid 8
sql_1  | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
api_1  | {"level":"warning","time":"2019-01-04T08:26:37.252038723Z","message":"Failed to connect to SQL database. Retrying in 2 seconds.","data":"dial tcp 10.19.84.4:3306: getsockopt: connection refused"}
es_1   | [2019-01-04T08:26:44,966][INFO ][o.e.n.Node               ] [qxFs6OU] started
ui_1   | 2019-01-04 08:26:31,710 INFO success: webui entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
sql_1  | To do so, start the server, then issue the following commands:
api_1  | {"level":"info","time":"2019-01-04T08:26:39.253442048Z","message":"Successfully connected to SQL database."}
es_1   | [2019-01-04T08:26:45,062][INFO ][o.e.g.GatewayService     ] [qxFs6OU] recovered [0] indices into cluster_state
sql_1  |
api_1  | {"level":"warning","time":"2019-01-04T08:26:44.409947762Z","message":"Failed to connect to ElasticSearch database. Retrying in %!s(int=2) seconds.","data":"health check timeout: Head http://es:9200: dial tcp 10.19.84.3:9200: getsockopt: connection refused: no Elasticsearch node available"}
es_1   | [2019-01-04T08:26:45,379][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [qxFs6OU] successfully created .ml-notifications index template
sql_1  | '/usr/bin/mysqladmin' -u root password 'new-password'
api_1  | {"level":"info","time":"2019-01-04T08:26:46.676902103Z","message":"Successfully connected to ElasticSearch database."}
es_1   | [2019-01-04T08:26:45,464][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [qxFs6OU] successfully created .ml-state index template
sql_1  | '/usr/bin/mysqladmin' -u root -h  password 'new-password'
api_1  | {"level":"info","time":"2019-01-04T08:26:46.720897383Z","message":"Put ES index lineitems.","data":{"acknowledged":true}}
es_1   | [2019-01-04T08:26:45,487][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [qxFs6OU] successfully created .ml-meta index template
sql_1  |
api_1  | {"level":"info","time":"2019-01-04T08:26:46.759298144Z","message":"Put ES index AnomaliesDetection.","data":{"acknowledged":true}}
es_1   | [2019-01-04T08:26:45,534][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [qxFs6OU] successfully created .ml-anomalies- index template
es_1   | [2019-01-04T08:26:45,783][INFO ][o.e.l.LicenseService     ] [qxFs6OU] license [b117f115-3038-4f87-bcea-817a1c3945cd] mode [trial] - valid
sql_1  | Alternatively you can run:
api_1  | {"level":"info","time":"2019-01-04T08:26:46.783406905Z","message":"Put ES index EC2Report.","data":{"acknowledged":true}}
sql_1  | '/usr/bin/mysql_secure_installation'
sql_1  |
api_1  | {"level":"info","time":"2019-01-04T08:26:46.80606293Z","message":"Put ES index EC2 Coverage Report.","data":{"acknowledged":true}}
sql_1  | which will also give you the option of removing the test
sql_1  | databases and anonymous user created by default.  This is
api_1  | {"level":"info","time":"2019-01-04T08:26:46.82978386Z","message":"Put ES index ElastiCache Report.","data":{"acknowledged":true}}
sql_1  | strongly recommended for production servers.
sql_1  |
sql_1  | See the MariaDB Knowledgebase at http://mariadb.com/kb or the
api_1  | {"level":"info","time":"2019-01-04T08:26:46.853230022Z","message":"Put ES index ESReport.","data":{"acknowledged":true}}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.875109694Z","message":"Put ES index rds-reports.","data":{"acknowledged":true}}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.896676091Z","message":"Put ES index account-plugins.","data":{"acknowledged":true}}
sql_1  | MySQL manual for more instructions.
api_1  | {"level":"info","time":"2019-01-04T08:26:46.919304328Z","message":"Put ES index LambdaReport.","data":{"acknowledged":true}}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.942740832Z","message":"Put ES index ReservedInstancesReport.","data":{"acknowledged":true}}
sql_1  |
api_1  | {"level":"info","time":"2019-01-04T08:26:46.964842393Z","message":"Put ES index rds-ri-reports.","data":{"acknowledged":true}}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965185801Z","message":"Started.","data":{"backendId":"c5bd6e79d172-e1b2acaf"}}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965366838Z","message":"Registered route /user."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965454595Z","message":"Registered route /user/viewer."}
sql_1  | Please report any problems at http://mariadb.org/jira
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965534201Z","message":"Registered route /user/login."}
sql_1  |
sql_1  | The latest information about MariaDB is available at http://mariadb.org/.
api_1  | {"level":"info","time":"2019-01-04T08:26:46.96561251Z","message":"Registered route /user/password/forgotten."}
sql_1  | You can find additional information about the MySQL part at:
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965694603Z","message":"Registered route /user/password/reset."}
sql_1  | http://dev.mysql.com
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965773453Z","message":"Registered route /aws/billrepository."}
sql_1  | Consider joining MariaDB's strong and vibrant community:
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965851138Z","message":"Registered route /aws/billrepositoryupdates."}
sql_1  | https://mariadb.org/get-involved/
api_1  | {"level":"info","time":"2019-01-04T08:26:46.965924772Z","message":"Registered route /aws."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966009327Z","message":"Registered route /aws/subaccount."}
sql_1  |
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966068011Z","message":"Registered route /aws/next."}
sql_1  | Database initialized
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966158534Z","message":"Registered route /aws/status."}
sql_1  | MySQL init process in progress...
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966233233Z","message":"Registered route /costs."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966290696Z","message":"Registered route /costs/anomalies."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 97 ...
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966382513Z","message":"Registered route /ec2."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966434998Z","message":"Registered route /ec2/unused."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966523785Z","message":"Registered route /costs/diff."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966604845Z","message":"Registered route /costs/tags/values."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966656487Z","message":"Registered route /costs/tags/keys."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Uses event mutexes
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966724751Z","message":"Registered route /plugins/results."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966806776Z","message":"Registered route /elasticache."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Compressed tables use zlib 1.2.8
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Using Linux native AIO
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966865929Z","message":"Registered route /elasticache/unused."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Number of pools: 1
api_1  | {"level":"info","time":"2019-01-04T08:26:46.966952045Z","message":"Registered route /es."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Using SSE2 crc32 instructions
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967031861Z","message":"Registered route /es/unused."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Completed initialization of buffer pool
sql_1  | 2019-01-04  8:26:31 139952870475520 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Highest supported file format is Barracuda.
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: 128 out of 128 rollback segments are active.
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967078057Z","message":"Registered route /lambda."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Creating shared tablespace for temporary tables
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967157257Z","message":"Registered route /rds."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: Waiting for purge to start
api_1  | {"level":"info","time":"2019-01-04T08:26:46.96723126Z","message":"Registered route /rds/unused."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967309101Z","message":"Registered route /reports."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967367537Z","message":"Registered route /report."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] InnoDB: 5.7.19 started; log sequence number 1619987
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967447021Z","message":"Registered route /s3/costs."}
sql_1  | 2019-01-04  8:26:31 139952364967680 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967512934Z","message":"Registered route /ec2/coverage."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] Plugin 'FEEDBACK' is disabled.
api_1  | {"level":"info","time":"2019-01-04T08:26:46.9675619Z","message":"Registered route /ri/ec2."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967640239Z","message":"Registered route /ri/rds."}
sql_1  | 2019-01-04  8:26:31 139952364967680 [Note] InnoDB: Buffer pool(s) load completed at 190104  8:26:31
api_1  | {"level":"info","time":"2019-01-04T08:26:46.96770609Z","message":"Registered route /user/share."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967755306Z","message":"Registered route /docs."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Warning] 'user' entry 'root@fba613fdadad' ignored in --skip-name-resolve mode.
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967839901Z","message":"Scheduled periodic tasks."}
api_1  | {"level":"info","time":"2019-01-04T08:26:46.967910918Z","message":"Listening on [::]:80."}
sql_1  | 2019-01-04  8:26:31 139953591863168 [Warning] 'proxies_priv' entry '@% root@fba613fdadad' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] Reading of all Master_info entries succeded
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] Added new Master_info '' to hash table
sql_1  | 2019-01-04  8:26:31 139953591863168 [Note] mysqld: ready for connections.
sql_1  | Version: '10.2.9-MariaDB-10.2.9+maria~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
sql_1  | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
sql_1  | GENERATED ROOT PASSWORD: quo9Nu8UCh8ahgeedaeL8moo7azek8uw
sql_1  | 2019-01-04  8:26:35 139952956409600 [Warning] 'proxies_priv' entry '@% root@fba613fdadad' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-04  8:26:35 139952956409600 [Warning] 'proxies_priv' entry '@% root@fba613fdadad' ignored in --skip-name-resolve mode.
sql_1  |
sql_1  | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/schema.sql
sql_1  |
sql_1  |
sql_1  | 2019-01-04  8:26:37 139952956106496 [Note] mysqld (unknown): Normal shutdown
sql_1  |
sql_1  | 2019-01-04  8:26:37 139952956106496 [Note] Event Scheduler: Purging the queue. 0 events
sql_1  | 2019-01-04  8:26:37 139952750835456 [Note] InnoDB: FTS optimize thread exiting.
sql_1  | 2019-01-04  8:26:37 139952956106496 [Note] InnoDB: Starting shutdown...
sql_1  | 2019-01-04  8:26:37 139952364967680 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
sql_1  | 2019-01-04  8:26:37 139952364967680 [Note] InnoDB: Buffer pool(s) dump completed at 190104  8:26:37
sql_1  | 2019-01-04  8:26:38 139952956106496 [Note] InnoDB: Shutdown completed; log sequence number 2323307
sql_1  | 2019-01-04  8:26:38 139952956106496 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
sql_1  | 2019-01-04  8:26:38 139952956106496 [Note] mysqld: Shutdown complete
sql_1  |
sql_1  |
sql_1  | MySQL init process done. Ready for start up.
sql_1  |
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] mysqld (mysqld 10.2.9-MariaDB-10.2.9+maria~jessie) starting as process 1 ...
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Uses event mutexes
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Compressed tables use zlib 1.2.8
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Using Linux native AIO
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Number of pools: 1
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Using SSE2 crc32 instructions
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Completed initialization of buffer pool
sql_1  | 2019-01-04  8:26:38 140426037094144 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
sql_1  | 2019-01-04  8:26:38 140426758686592 [Note] InnoDB: Highest supported file format is Barracuda.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: 128 out of 128 rollback segments are active.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: Creating shared tablespace for temporary tables
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: Waiting for purge to start
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] InnoDB: 5.7.19 started; log sequence number 2323307
sql_1  | 2019-01-04  8:26:39 140425868326656 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] Plugin 'FEEDBACK' is disabled.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] Server socket created on IP: '::'.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Warning] 'proxies_priv' entry '@% root@fba613fdadad' ignored in --skip-name-resolve mode.
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] Reading of all Master_info entries succeded
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] Added new Master_info '' to hash table
sql_1  | 2019-01-04  8:26:39 140426758686592 [Note] mysqld: ready for connections.
sql_1  | Version: '10.2.9-MariaDB-10.2.9+maria~jessie'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
sql_1  | 2019-01-04  8:26:39 140425868326656 [Note] InnoDB: Buffer pool(s) load completed at 190104  8:26:39
es_1   | [2019-01-04T08:26:51,780][INFO ][o.e.c.m.MetaDataCreateIndexService] [qxFs6OU] [.monitoring-es-6-2019.01.04] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]
es_1   | [2019-01-04T08:26:51,946][INFO ][o.e.c.m.MetaDataCreateIndexService] [qxFs6OU] [.watches] creating index, cause [auto(bulk api)], templates [watches], shards [1]/[1], mappings [watch]
es_1   | [2019-01-04T08:26:52,173][INFO ][o.e.c.m.MetaDataMappingService] [qxFs6OU] [.watches/xXa5S8_cTD2ebHqqF3fo-w] update_mapping [watch]
es_1   | [2019-01-04T08:27:14,940][INFO ][o.e.c.r.a.DiskThresholdMonitor] [qxFs6OU] low disk watermark [85%] exceeded on [qxFs6OUqTVCOiQXlkExiig][qxFs6OU][/usr/share/elasticsearch/data/nodes/0] free: 4.3gb[14.8%], replicas will not be assigned to this node

Here is the running containers.

ocker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                      NAMES
c5bd6e79d172        trackit_api                                           "/main -sql-address=…"   3 minutes ago       Up 2 minutes        0.0.0.0:8080->80/tcp                       trackit_api_1
fba613fdadad        trackit_sql                                           "docker-entrypoint.s…"   3 minutes ago       Up 3 minutes        127.0.0.1:3306->3306/tcp                   trackit_sql_1
e31ff5ebc91d        docker.elastic.co/elasticsearch/elasticsearch:5.5.3   "/bin/bash bin/es-do…"   3 minutes ago       Up 3 minutes        127.0.0.1:9200->9200/tcp, 9300/tcp         trackit_es_1
ea0e400bc3e1        trackit_ui                                            "/usr/bin/supervisord"   3 minutes ago       Up 3 minutes        0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   trackit_ui_1

Still i am having the same problem, while registering, login or even the forgot password link.

@thibautcornolti
Copy link
Member

I see you still have these errors

WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.

For now, we'll ignore them, to understand why you can't login.

Go to http://127.0.0.1/, you'll see the login page.
Then, display the network page in the developer tools (Press F12 or Ctrl + Shift + I).
Try to create an account. You'll see a request with an error, write here the details.

@sathyajackson
Copy link
Author

The below snip is what i see when i try to register a new user using my gmail email address

image

@thibautcornolti
Copy link
Member

Please click on the last one and screen again

@sathyajackson
Copy link
Author

I think i got the issue sorted out. It was beacuse of a proxy server in the network which was causing the problem. I sorted it out and now i can able to register.

But why am i still getting this warning.

[ warning ]

WARNING: The AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AWS_SESSION_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.

@thibautcornolti
Copy link
Member

thibautcornolti commented Jan 4, 2019 •
edited
Loading

If

  • you started TrackIt with start.sh
  • you have a default profile in AWS
  • the following command works aws --profile default sts get-caller-identity
  • you can get results with ./trackit-server/scripts/awsenv default env | grep 'AWS_' | cut -f1 -d"="

I'm not sure why you're still getting this

@sathyajackson
Copy link
Author

Just want to understand, why we need to provide read-only access to an external account to have this working?

@thibautcornolti
Copy link
Member

TrackIt needs an AWS account to assume roles (https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
Roles are asked in the UI and used to fetch bills and other data from the AWS account you want to monitor.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@thibautcornolti thibautcornolti

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@thibautcornolti @lfrancois @sathyajackson