Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Migration log: reduce number of entries and use microtime to avoid chances of conflicts #259

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

brainexe
Copy link
Member

@brainexe brainexe commented Oct 11, 2023

I got some very rare errors during igvm migrate when two migrations of the same source hypervisor finished at the same second with the same CPU value:


INFO: igvm.commands: Picked hv-destination.local as destination Hypervisor
[hv-destination.local] sudo: while [ ! -L '/dev/xen-data/469788_vm-name' ]; do sleep 1; done
INFO: igvm.kvm: Starting online migration of vm vm-name from hv-source.local to hv-destination.local
INFO: igvm.kvm: Waiting for migration stats to show up
INFO: igvm.kvm: Migration progress: disk 0% 0/0MiB, memory 0% 0/0MiB,
INFO: igvm.kvm: Migration progress: disk 0% 0/0MiB, memory 0% 0/0MiB,
INFO: igvm.kvm: Migration progress: disk 2% 305/17408MiB, memory 0% 0/0MiB,
...
INFO: igvm.kvm: Migration progress: disk 100% 17410/17410MiB, memory 93% 2869/3073MiB,
libvirt: Domain Config error : Requested operation is not valid: domain is not running
INFO: igvm.kvm: Awaiting migration to finish
INFO: igvm.kvm: Migration finished
INFO: igvm.transaction: Rolling back transaction
libvirt:  error : internal error: Child process (/sbin/lvchange -aln /dev/xen-data/469788_vm-name) unexpected exit status 5:   Logical volume xen-data/469788_vm-name in use.
WARNING: igvm.transaction: Rollback action "destroy storage" failed: internal error: Child process (/sbin/lvchange -aln /dev/xen-data/469788_vm-name) unexpected exit status 5:   Logical volume xen-data/469788_vm-name in use.

Disconnecting from hv-destination.local... done.
Traceback (most recent call last):
  File "/usr/bin/igvm", line 33, in <module>
    sys.exit(load_entry_point('igvm==2.2.0', 'console_scripts', 'igvm')())
  File "/usr/lib/python3/dist-packages/igvm/cli.py", line 496, in main
    args.pop('func')(**args)
  File "/usr/lib/python3/dist-packages/igvm/host.py", line 25, in decorator
    return fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/igvm/commands.py", line 531, in vm_migrate
    previous_hypervisor.log_migration(_vm, '-')
  File "/usr/lib/python3/dist-packages/igvm/hypervisor.py", line 1157, in log_migration
    self.dataset_obj.commit()
  File "/usr/lib/python3/dist-packages/adminapi/dataset.py", line 488, in commit
    _handle_exception(result)
  File "/usr/lib/python3/dist-packages/adminapi/dataset.py", line 574, in _handle_exception
    raise exception_class(result['message'])
adminapi.exceptions.DatasetError: {'__all__': ['Server string attribute with this Server, Attribute and Value already exists.']}

Note:

  • adapt the regexp for the SA attribute to something like \A([0-9]{10}|\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d+)\s[-|+][0-9]{1,3}\Z
  • link needed innogames/serveradmin changes

igvm/hypervisor.py Outdated Show resolved Hide resolved
@@ -518,7 +518,7 @@ def test_igvm_migration_log(self, performance_value, mock_time):

src_hv = self.vm.hypervisor.dataset_obj['hostname']
cpu_usage_vm_src = self.vm.hypervisor.estimate_vm_cpu_usage(self.vm)
timestamp = 1234567890
timestamp = 1234567890000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this test timestamp be updated to the new format too, in case we ever restrict in Serveradmin?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants