Grafana crashing

[knmorgan]

I had this successfully running on my Synology previously, but re-initialized the volume and cannot get it to work again. When following the directions on this repo I get the following logs from supervisord ad infinitum.

2022-12-20 04:11:10,233 INFO spawned: 'grafana' with pid 5185
2022-12-20 04:11:11,256 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-12-20 04:11:11,731 INFO exited: grafana (exit status 1; not expected)
2022-12-20 04:11:11,731 INFO reaped unknown pid 5196
2022-12-20 04:11:11,733 INFO spawned: 'grafana' with pid 5205
2022-12-20 04:11:12,756 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-12-20 04:11:12,757 INFO reaped unknown pid 5216
2022-12-20 04:11:12,767 INFO exited: grafana (exit status 1; not expected)

Looking in the grafana log I see complaints about the database being locked initially:

logger=settings t=2022-12-20T04:05:14.515925475Z level=info msg="Starting Grafana" version=9.1.3 commit=f3f9a818c2 branch=HEAD compiled=2022-09-05T15:59:46Z
logger=settings t=2022-12-20T04:05:14.516336991Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-12-20T04:05:14.516396709Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2022-12-20T04:05:14.516425871Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2022-12-20T04:05:14.516454471Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2022-12-20T04:05:14.516483262Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2022-12-20T04:05:14.516511072Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2022-12-20T04:05:14.516539953Z level=info msg="Config overridden from Environment variable" var="GF_DATABASE_TYPE=sqlite3"
logger=settings t=2022-12-20T04:05:14.516568518Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2022-12-20T04:05:14.516595898Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2022-12-20T04:05:14.516622801Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2022-12-20T04:05:14.516649685Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2022-12-20T04:05:14.51667711Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2022-12-20T04:05:14.516704603Z level=info msg="App mode production"
logger=sqlstore t=2022-12-20T04:05:14.516869675Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2022-12-20T04:05:14.516922141Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rwxr-xr-x expected=-rw-r-----
logger=migrator t=2022-12-20T04:05:15.021613642Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7"
logger=settings t=2022-12-20T04:05:15.574853239Z level=info msg="Starting Grafana" version=9.1.3 commit=f3f9a818c2 branch=HEAD compiled=2022-09-05T15:59:46Z
logger=settings t=2022-12-20T04:05:15.575354207Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-12-20T04:05:15.575429981Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2022-12-20T04:05:15.575478012Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2022-12-20T04:05:15.57550559Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2022-12-20T04:05:15.575532263Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2022-12-20T04:05:15.575559553Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2022-12-20T04:05:15.575587645Z level=info msg="Config overridden from Environment variable" var="GF_DATABASE_TYPE=sqlite3"
logger=settings t=2022-12-20T04:05:15.57561536Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2022-12-20T04:05:15.575641907Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2022-12-20T04:05:15.57566846Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2022-12-20T04:05:15.575694686Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2022-12-20T04:05:15.575721178Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2022-12-20T04:05:15.57574836Z level=info msg="App mode production"
logger=sqlstore t=2022-12-20T04:05:15.575924431Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2022-12-20T04:05:15.575976855Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rwxr-xr-x expected=-rw-r-----
logger=migrator t=2022-12-20T04:05:15.588282961Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7"
logger=migrator t=2022-12-20T04:05:16.419610267Z level=error msg="alert migration failure: could not get migration log" error="failed to check table existence: database is locked"
logger=migrator t=2022-12-20T04:05:16.548090565Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7"

This lasts about 5 minutes, after which it no longer complains, but gets stuck in a loop without any apparent errors:

logger=settings t=2022-12-20T04:15:12.661900194Z level=info msg="Starting Grafana" version=9.1.3 commit=f3f9a818c2 branch=HEAD compiled=2022-09-05T15:59:46Z
logger=settings t=2022-12-20T04:15:12.662414423Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-12-20T04:15:12.662468342Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2022-12-20T04:15:12.662495231Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2022-12-20T04:15:12.662522201Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2022-12-20T04:15:12.662551291Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2022-12-20T04:15:12.662578604Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2022-12-20T04:15:12.662605697Z level=info msg="Config overridden from Environment variable" var="GF_DATABASE_TYPE=sqlite3"
logger=settings t=2022-12-20T04:15:12.662632568Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2022-12-20T04:15:12.662658035Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2022-12-20T04:15:12.662683229Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2022-12-20T04:15:12.662718377Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2022-12-20T04:15:12.66274584Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2022-12-20T04:15:12.6627715Z level=info msg="App mode production"
logger=sqlstore t=2022-12-20T04:15:12.662938787Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2022-12-20T04:15:12.662989201Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rwxr-xr-x expected=-rw-r-----
logger=migrator t=2022-12-20T04:15:12.71598385Z level=info msg="Starting DB migrations"
logger=migrator t=2022-12-20T04:15:12.734386423Z level=info msg="migrations completed" performed=0 skipped=443 duration=1.184758ms

Owner of /var/lib/grafana is grafana:grafana. The telegraf service is running, but reporting errors writing to influxdb.

2022-12-20T04:05:29Z I! Starting Telegraf 1.16.3
2022-12-20T04:05:29Z I! Loaded inputs: cpu disk diskio kernel mem processes snmp statsd swap syslog system
2022-12-20T04:05:29Z I! Loaded aggregators:
2022-12-20T04:05:29Z I! Loaded processors:
2022-12-20T04:05:29Z I! Loaded outputs: influxdb
2022-12-20T04:05:29Z I! Tags enabled: host=telegraf-influxdb-grafana
2022-12-20T04:05:29Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"telegraf-influxdb-grafana", Flush Interval:10s
2022-12-20T04:05:29Z I! [inputs.statsd] UDP listening on "[::]:8125"
2022-12-20T04:05:29Z I! [inputs.statsd] Started the statsd service on ":8125"
2022-12-20T04:05:44Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-12-20T04:05:44Z E! [agent] Error writing to outputs.influxdb: could not write any address

Any idea what's happening? This was tested with the same result both on latest and develop.

[pad92]

duplicate with #8