Hello.
I've got a 13.2p9/amd64 server where I have a jail dedicated to
ElasticSearch+Kibana and I'm having troubles starting and stopping the
latter.
# pkg info | grep -E "(elastic|kibana)"
elasticsearch8-8.11.3 Distributed, RESTful search and analytics engine
kibana8-8.11.3 Browser based analytics and search interface to
ElasticSearch
py39-elasticsearch-7.17.9 Official Python low-level client for
Elasticsearch
py39-elasticsearch-dsl-7.3.0 High level Python client for Elasticsearch
# cat /etc/rc.conf
#cron_enable="NO"
kibana_enable="YES"
kibana_syslog_output_enable="YES"
elasticsearch_enable="YES"
First problem
When I start the jail (with "ezjail-admin start") Elastic starts, but
Kibana does not.
syslog shows:
[2024-02-03T16:14:21.861+01:00][INFO ][root] Kibana is starting
[2024-02-03T16:14:21.939+01:00][INFO ][root] Kibana is shutting down
[2024-02-03T16:14:21.939+01:00][FATAL][root] Reason: EACCES: permission denied,
open '/var/run/kibana.pid'
Error: EACCES: permission denied, open '/var/run/kibana.pid'
FATAL Error: EACCES: permission denied, open '/var/run/kibana.pid'
Same happens if I try "service kibana start" afterwards from inside the
jail.
I can get around this with:
# touch /var/run/kibana.pid
# chown www /var/run/kibana.pid
# service kibana start
However this must be done manually everytime I need to restart the
server or jail.
Second problem
When I try to stop Kibana, it just hangs indefinitely:
# service kibana stop
Stopping kibana.
Waiting for PIDS: 28179
syslog reports:
[2024-02-03T16:20:22.309+01:00][INFO ][root] SIGTERM received - initiating
shutdown
[2024-02-03T16:20:22.309+01:00][INFO ][root] Kibana is shutting down
[2024-02-03T16:20:22.311+01:00][INFO ][plugins-system.standard] Stopping all
plugins.
[2024-02-03T16:20:22.314+01:00][INFO
][plugins.monitoring.monitoring.kibana-monitoring] Monitoring stats collection
is stopped
[2024-02-03T16:20:22.323+01:00][INFO ][plugins-system.standard] All plugins
stopped.
[2024-02-03T16:20:22.324+01:00][WARN ][environment] Detected an unhandled
Promise rejection: Error: EACCES: permission denied, unlink
'/var/run/kibana.pid'
at unlinkSync (node:fs:1883:3)
at process.<anonymous>
(/usr/local/www/kibana8/node_modules/@kbn/core-environment-server-internal/src/write_pid_file.js:49:24)
at process.<anonymous>
(/usr/local/www/kibana8/node_modules/lodash/before.js:31:21)
at Object.onceWrapper (node:events:632:26)
at process.emit (node:events:529:35)
at process.emit
(/usr/local/www/kibana8/node_modules/source-map-support/source-map-support.js:516:21)
at process.exit (node:internal/process/per_thread:192:15)
at Root.onRootShutdown [as onShutdown]
(/usr/local/www/kibana8/node_modules/@kbn/core-root-server-internal/src/bootstrap.js:136:11)
at Root.shutdown
(/usr/local/www/kibana8/node_modules/@kbn/core-root-server-internal/src/root/index.js:96:12)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
[2024-02-03T16:20:23.026+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:26.024+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:29.024+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:32.023+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:35.022+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:38.021+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:41.021+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:44.021+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:47.022+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:50.022+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:53.023+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:56.022+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:20:59.021+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:21:02.021+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:21:05.021+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
NoLivingConnectionsError: There are no living connections
[2024-02-03T16:21:08.022+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:21:11.023+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:21:14.022+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:21:17.021+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
[2024-02-03T16:21:20.020+01:00][ERROR][plugins.taskManager] Failed to poll for
work: NoLivingConnectionsError: There are no living connections
This goes on until I kill -9 the process.
This is particularly problematic in case power goes missing and the UPS
tries to shutdown the server, which will hang, instead.
Any hint?
bye & Thanks
av.
