hosted-engine deploy fails at lockspace volume initialization (FC Storage) #20
Description
OS: CentOS 8
oVirt: nightly.
hosted-engine deployment fails at the step: Initialize lockspace volume
(i think, at hosted-engine --reinitialize-lockspace :) )
[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"], "delta": "0:00:00.315076", "end": "2023-11-25 00:31:19.175840", "msg": "non-zero return code", "rc": 1, "start": "2023-11-25 00:31:18.860764", "stderr": "Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py", line 30, in <module>
ha_cli.reset_lockspace(force)
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 286, in reset_lockspace
stats = broker.get_stats_from_storage()
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 148, in get_stats_from_storage
result = self._proxy.get_stats()
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__
return sel$.__send(self.__name, args)
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in __request
verbose=self.__verbose
File "/usr/lib64/pyth$n3.6/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib64/python3.6/xmlrpc/$lient.py", line 1166, in single_request
http_conn = self.send_request(host, handler, request_body, verbose)
File "/usr/lib64/python3.6/xmlrpc/cli$nt.py", line 1279, in send_request
self.send_content(connection, request_body)
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1309, in send_$ontent
connection.endheaders(request_body)
File "/usr/lib64/python3.6/http/client.py", line 1268, in endheaders
self._send_output(message_bo$y, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_output
self.send(msg)
File "/usr/lib64/pyth$n3.6/http/client.py", line 982, in send
self.connect()
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 76, in connect
self.sock.connect(base64.b16decode(self.host))
FileNotFoundError: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most re$ent call last):", " File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main", " "__main__", mod_spec)", " File "/usr/lib64/python3$6/runpy.py", line 85, in _run_code", " exec(code, run_globals)", " File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lock$pace.py", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py"$ line 286, in reset_lockspace", " stats = broker.get_stats_from_storage()", " File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/broker$ink.py", line 148, in get_stats_from_storage", " result = self._proxy.get_stats()", " File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __$all__", " return self.__send(self.__name, args)", " File "/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in __request", " verbose=self.__verbo$e", " File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request", " return self.single_request(host, handler, request_body, verbose)", " F$le "/usr/lib64/python3.6/xmlrpc/client.py", line 1166, in single_request", " http_conn = self.send_request(host, handler, request_body, verbose)", " $ile "/usr/lib64/python3.6/xmlrpc/client.py", line 1279, in send_request", " self.send_content(connection, request_body)", " File "/usr/lib64/python3$6/xmlrpc/client.py", line 1309, in send_content", " connection.endheaders(request_body)", " File "/usr/lib64/python3.6/http/client.py", line 1268, i$ endheaders", " self._send_output(message_body, encode_chunked=encode_chunked)", " File "/usr/lib64/python3.6/http/client.py", line 1044, in _send_ou$put", " self.send(msg)", " File "/usr/lib64/python3.6/http/client.py", line 982, in send", " self.connect()", " File "/usr/lib/python3.6/site-pa$kages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 76, in connect", " self.sock.connect(base64.b16decode(self.host))", "FileNotFoundError: [Errno 2] No\r
such file or directory"], "stdout": "", "stdout_lines": []}
it's trying to open the /var/run/ovirt-hosted-engine-ha/broker.socket file which doesn't exist.
even if the ovirt-ha-broker is running.
In the ovirt-ha-broker logfile i found some error messages:
StatusStorageThread::ERROR::2023-11-25 01:20:41,001::storage_broker::167::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(get_raw_stats) Corrupted metadata from /run/vdsm/storage/b1c01f9c-0534-4e24-980a-f57a840469cd/a58fa5c6-03ec-41b7-97c3-9ac84eab5362/c1f5a6ce-9426-4da0-af04-23e21fdec67a
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 163, in get_raw_stats
data = bdata.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 1048576: invalid start byte
StatusStorageThread::ERROR::2023-11-25 01:20:41,002::status_broker::98::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run) Failed to read state.
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 163, in get_raw_stats
data = bdata.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 1048576: invalid start byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 94, in run
self._storage_broker.get_raw_stats()
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 169, in get_raw_stats
.format(str(e)))
ovirt_hosted_engine_ha.lib.exceptions.RequestError: Corrupted read metadata: 'utf-8' codec can't decode byte 0xa0 in position 1048576: invalid start byte
StatusStorageThread::ERROR::2023-11-25 01:20:41,002::status_broker::70::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(trigger_restart) Trying to restart the broker
/run/vdsm/storage/b1c01f9c-0534-4e24-980a-f57a840469cd/a58fa5c6-03ec-41b7-97c3-9ac84eab5362/c1f5a6ce-9426-4da0-af04-23e21fdec67a is the symlink to /dev/b1c01f9c-0534-4e24-980a-f57a840469cd/c1f5a6ce-9426-4da0-af04-23e21fdec67a which is
b1c01f9c--0534--4e24--980a--f57a840469cd-c1f5a6ce--9426--4da0--af04--23e21fdec67a
│ 253:24 0 128M 0 lvm
└─3600c0ff00012158dd030616501000000 253:12 0 18.2T 0 mpath
└─sdb 8:16 0 18.2T 0 disk