Na heel wat jaren bescheiden Domoticz-setup (influxddb & grafana) op een RPi te hebben gedraaid ben ik nu aan het proberen een HA-omgeving op te bouwen. De basic setup draait in een VM op m'n Synology conform the installatiehandleiding van HA zelf, dat ging ook vrij snel. De VM heeft 2 CPU's, 3Gb ram en 332Gb schijfruimte. Geïnstalleerde add-ons zijn Deconz, Grafana, Influzdb en Studio Code Server.
Maar...met het opbouwen van Influxdb en met name Grafana stokt het. De Grafana-addon gedraagt zich onvoorspelbaar met opstarten/herstarten. Soms in 1 keer probleemloos, soms na een keer herstarten pas, soms blijft het misgaan. Sowieso gaat het opstarten van alles, de HA én addons naar mijn idee wat aan de trage kan.
Ik kan vooralsnog geen vinger op de zere plek leggen, zelfs niet met behulp van logfiles.
Dit is een recente grafana-logfile:
logger=modules t=2023-12-02T12:56:40.340314379+01:00 level=info msg="All modules stopped" modules="[http-server secret-migrator provisioning background-services]"
Error: ✗ *rendering.RenderingService run error: timeout while waiting for plugin to start
2023/12/02 12:56:44 [error] 503#503: *26 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: a0d7b954-grafana, request: "GET /api/health HTTP/1.1", upstream: "http://127.0.0.1:3000/api/health", host: "127.0.0.1:1337"
s6-supervise grafana: warning: finish script lifetime reached maximum value - sending it a SIGKILL
[12:56:49] INFO: Starting Grafana...
Grafana server is running with elevated privileges. This is not recommended
logger=settings t=2023-12-02T12:57:01.875447842+01:00 level=info msg="Starting Grafana" version=10.1.2 commit=8e428858dd branch=HEAD compiled=2023-09-18T22:41:04+02:00
logger=settings t=2023-12-02T12:57:01.876383185+01:00 level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2023-12-02T12:57:01.876419501+01:00 level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2023-12-02T12:57:01.876456626+01:00 level=info msg=Target target=[all]
logger=settings t=2023-12-02T12:57:01.876482275+01:00 level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2023-12-02T12:57:01.87649593+01:00 level=info msg="Path Data" path=/data
logger=settings t=2023-12-02T12:57:01.876512054+01:00 level=info msg="Path Logs" path=/var/logs/grafana
logger=settings t=2023-12-02T12:57:01.876524646+01:00 level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2023-12-02T12:57:01.876536987+01:00 level=info msg="Path Provisioning" path=/usr/share/grafana/conf/provisioning
logger=settings t=2023-12-02T12:57:01.876550111+01:00 level=info msg="App mode production"
logger=sqlstore t=2023-12-02T12:57:01.877799539+01:00 level=info msg="Connecting to DB" dbtype=sqlite3
logger=migrator t=2023-12-02T12:57:01.969149051+01:00 level=info msg="Starting DB migrations"
logger=migrator t=2023-12-02T12:57:02.006990288+01:00 level=info msg="migrations completed" performed=0 skipped=493 duration=1.438422ms
logger=secrets t=2023-12-02T12:57:02.009595612+01:00 level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=plugin.loader t=2023-12-02T12:57:02.187473461+01:00 level=warn msg="Plugin missing module.js" pluginID=input warning="Missing module.js, If you loaded this plugin from git, make sure to compile it."
logger=plugin.loader t=2023-12-02T12:57:02.187640313+01:00 level=info msg="Plugin registered" pluginID=input
logger=plugin.loader t=2023-12-02T12:57:07.697591901+01:00 level=info msg="Plugin registered" pluginID=grafana-image-renderer
logger=query_data t=2023-12-02T12:57:07.702076362+01:00 level=info msg="Query Service initialization"
logger=live.push_http t=2023-12-02T12:57:07.710235856+01:00 level=info msg="Live Push Gateway initialization"
logger=infra.usagestats.collector t=2023-12-02T12:57:14.858456168+01:00 level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=modules t=2023-12-02T12:57:14.859112712+01:00 level=info msg=initialising module=http-server
logger=modules t=2023-12-02T12:57:14.860187414+01:00 level=info msg=initialising module=secret-migrator
logger=http.server t=2023-12-02T12:57:14.865624369+01:00 level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl=/api/hassio_ingress/nbXBeFeJZVVGV36vNui9KcBb0ENghAR8c2dQ_Ul2zU0 socket=
logger=modules t=2023-12-02T12:57:22.522075861+01:00 level=info msg=initialising module=provisioning
logger=provisioning.alerting t=2023-12-02T12:57:22.524903675+01:00 level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2023-12-02T12:57:22.524944743+01:00 level=info msg="finished to provision alerting"
logger=modules t=2023-12-02T12:57:22.525770192+01:00 level=info msg=initialising module=background-services
logger=modules t=2023-12-02T12:57:22.527035429+01:00 level=info msg="All modules healthy" modules="[secret-migrator provisioning background-services http-server]"
logger=ngalert.state.manager t=2023-12-02T12:57:22.527361085+01:00 level=info msg="Warming state cache for startup"
logger=grafanaStorageLogger t=2023-12-02T12:57:22.529703966+01:00 level=info msg="storage starting"
logger=grafana.update.checker t=2023-12-02T12:57:22.716537272+01:00 level=info msg="Update check succeeded" duration=181.238187ms
logger=ngalert.state.manager t=2023-12-02T12:57:22.777793049+01:00 level=info msg="State cache has been initialized" states=0 duration=250.398202ms
logger=ngalert.scheduler t=2023-12-02T12:57:22.781543099+01:00 level=info msg="Starting scheduler" tickInterval=10s
logger=ticker t=2023-12-02T12:57:22.782738534+01:00 level=info msg=starting first_tick=2023-12-02T12:57:30+01:00
logger=ngalert.multiorg.alertmanager t=2023-12-02T12:57:22.783217006+01:00 level=info msg="Starting MultiOrg Alertmanager"
logger=background-services-runner t=2023-12-02T12:58:22.546181701+01:00 level=error msg="Stopped background service" service=*rendering.RenderingService reason="timeout while waiting for plugin to start"
logger=tracing t=2023-12-02T12:58:22.547080749+01:00 level=info msg="Closing tracing"
logger=ticker t=2023-12-02T12:58:22.548476914+01:00 level=info msg=stopped last_tick=2023-12-02T12:58:20+01:00
logger=plugin.grafana-image-renderer t=2023-12-02T12:58:22.549045585+01:00 level=warn msg="plugin failed to exit gracefully"
logger=plugin.grafana-image-renderer t=2023-12-02T12:58:22.557658915+01:00 level=error msg="plugin process exited" path=/var/lib/grafana/plugins/grafana-image-renderer/plugin_start_linux_amd64 pid=842 error="signal: killed"
logger=modules t=2023-12-02T12:58:22.561167511+01:00 level=warn msg="module failed with error" module=background-services err="*rendering.RenderingService run error: timeout while waiting for plugin to start"
logger=modules t=2023-12-02T12:58:22.561369482+01:00 level=info msg="Awaiting services to be stopped..."
logger=modules t=2023-12-02T12:58:22.56155287+01:00 level=info msg="module stopped" module=provisioning
logger=modules t=2023-12-02T12:58:22.561744165+01:00 level=info msg="module stopped" module=secret-migrator
logger=modules t=2023-12-02T12:58:22.561931477+01:00 level=info msg="module stopped" module=http-server
logger=modules t=2023-12-02T12:58:22.562037536+01:00 level=error msg="Module failed" module=background-services err="*rendering.RenderingService run error: timeout while waiting for plugin to start"
logger=modules t=2023-12-02T12:58:22.562058391+01:00 level=info msg="All modules stopped" modules="[background-services http-server secret-migrator provisioning]"
Error: ✗ *rendering.RenderingService run error: timeout while waiting for plugin to start
[12:58:24] INFO: Service Grafana exited with code 1 (by signal 0)
[12:58:29] INFO: Starting Grafana...
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service nginx: stopping
[12:58:35] INFO: Service NGINX exited with code 0 (by signal 0)
s6-rc: info: service nginx successfully stopped
s6-rc: info: service init-nginx: stopping
s6-rc: info: service grafana: stopping
s6-rc: info: service init-nginx successfully stopped
Grafana server is running with elevated privileges. This is not recommended
logger=settings t=2023-12-02T12:58:35.483282249+01:00 level=info msg="Starting Grafana" version=10.1.2 commit=8e428858dd branch=HEAD compiled=2023-09-18T22:41:04+02:00
logger=settings t=2023-12-02T12:58:35.484437932+01:00 level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2023-12-02T12:58:35.484474228+01:00 level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2023-12-02T12:58:35.484497696+01:00 level=info msg=Target target=[all]
logger=settings t=2023-12-02T12:58:35.486035783+01:00 level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2023-12-02T12:58:35.486078895+01:00 level=info msg="Path Data" path=/data
logger=settings t=2023-12-02T12:58:35.486094398+01:00 level=info msg="Path Logs" path=/var/logs/grafana
logger=settings t=2023-12-02T12:58:35.486234955+01:00 level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2023-12-02T12:58:35.486252779+01:00 level=info msg="Path Provisioning" path=/usr/share/grafana/conf/provisioning
logger=settings t=2023-12-02T12:58:35.486276812+01:00 level=info msg="App mode production"
logger=sqlstore t=2023-12-02T12:58:35.488841173+01:00 level=info msg="Connecting to DB" dbtype=sqlite3
[12:58:35] INFO: Service Grafana exited with code 256 (by signal 15)
s6-rc: info: service grafana successfully stopped
s6-rc: info: service init-grafana: stopping
s6-rc: info: service memcached: stopping
s6-rc: info: service init-grafana successfully stopped
logger=migrator t=2023-12-02T12:58:35.573118112+01:00 level=info msg="Starting DB migrations"
logger=migrator t=2023-12-02T12:58:35.593576972+01:00 level=info msg="migrations completed" performed=0 skipped=493 duration=1.328245ms
logger=secrets t=2023-12-02T12:58:35.595756872+01:00 level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=plugin.loader t=2023-12-02T12:58:35.744836195+01:00 level=warn msg="Plugin missing module.js" pluginID=input warning="Missing module.js, If you loaded this plugin from git, make sure to compile it."
logger=plugin.loader t=2023-12-02T12:58:35.744900706+01:00 level=info msg="Plugin registered" pluginID=input
[12:58:36] INFO: Service memcached exited with code 0 (by signal 0)
s6-rc: info: service memcached successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service base-addon-timezone: stopping
s6-rc: info: service base-addon-log-level: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service base-addon-log-level successfully stopped
s6-rc: info: service base-addon-timezone successfully stopped
s6-rc: info: service base-addon-banner: stopping
s6-rc: info: service base-addon-banner successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
Na een herstart van de VM (zonder aanpassingen aan Grafana) wil de addon gewoon helemaal niet meer starten.
Ik kan nu wel een losse Grafana-container op mn Synology draaien, maar alles in 1 VM is natuurlijk wel zo handig.
Zijn er ervaren HA-gebruikers hier die mij de juiste richting op kunnen wijzen om dit te verhelpen?