Skip to content
This repository has been archived by the owner on Jun 28, 2024. It is now read-only.

CCv0: improve debugability of SEV/SNP tests #5709

Closed
wants to merge 2 commits into from
Closed

CCv0: improve debugability of SEV/SNP tests #5709

wants to merge 2 commits into from

Conversation

wainersm
Copy link
Contributor

Let's print debug information on the SEV/SNP tests as currently we are left clueless about some fails.

Fixes #5708

wainersm added 2 commits July 12, 2023 10:50
@wainersm
kubernetes: system logs on SEV tests's teardown()
b2c69b2 
The SEV and SNP tests sometimes fail on CI and it is hard to determine
what went wrong because they don't print debug messages. This changed
the tests so that they behave like many others: print the system's logs
since the test's begin time.

Fixes #5708
Signed-off-by: Wainer dos Santos Moschetta <[email protected]>
@wainersm
kubernetes: move clean up routines to teardown()
565a630 
The setup() function of SEV and SNP tests execute some clean up routines
to get rid of resources left by the previous test. In reality the good
practice is to have the test remove its used resources on teardown.

Signed-off-by: Wainer dos Santos Moschetta <[email protected]>
@wainersm
Copy link
Contributor Author

/test-snp

@katacontainersbot katacontainersbot added the size/small Small and simple task label Jul 12, 2023
stevenhorsman
stevenhorsman approved these changes Jul 12, 2023
View reviewed changes
Copy link
Member

@stevenhorsman stevenhorsman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @wainersm

@wainersm
Copy link
Contributor Author

/test-sev

@wainersm
Copy link
Contributor Author

I ran the SNP test in a machine without SNP just to show how it will print the journalctl messages:

[init.sh:354] INFO: Create kata RuntimeClass resource
runtimeclass.node.k8s.io/kata created
[init.sh:34] INFO: Taint 'NoSchedule' is found. Untaint the node so pods can be scheduled.
node/ubuntu untainted
INFO: Run tests
1..1
not ok 1 [cc][kubernetes][containerd][snp] Test SNP unencrypted container launch success
# (from function `kubernetes_wait_for_pod_ready_state' in file lib.sh, line 42,
#  in test file confidential/snp.bats, line 87)
#   `kubernetes_wait_for_pod_ready_state "$pod_name" 20' failed
# service/snp-unencrypted created
# deployment.apps/snp-unencrypted created
# error: timed out waiting for the condition on pods/snp-unencrypted-68dd975f64-qjb27
# -- Kata logs:
# -- Logs begin at Wed 2023-07-12 14:21:58 UTC, end at Wed 2023-07-12 17:16:20 UTC. --
# Jul 12 17:16:01 ubuntu kata[186134]: time="2023-07-12T17:16:01.002913021Z" level=info msg="loaded configuration" file=/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-snp.toml format=TOML name=containerd-shim-v2 pid=186134 sandbox=3a2de150104295fa6cbec94e50548807017821fb8cdb79ea34ab9b5dd90ffdb7 source=katautils
# Jul 12 17:16:01 ubuntu kata[186134]: time="2023-07-12T17:16:01.003375524Z" level=info msg="IOMMUPlatform is disabled by default." name=containerd-shim-v2 pid=186134 sandbox=3a2de150104295fa6cbec94e50548807017821fb8cdb79ea34ab9b5dd90ffdb7 source=katautils
# Jul 12 17:16:01 ubuntu kata[186134]: time="2023-07-12T17:16:01.003852647Z" level=info msg="shm-size detected: 67108864" source=virtcontainers subsystem=oci
# Jul 12 17:16:01 ubuntu kata[186134]: time="2023-07-12T17:16:01.004742791Z" level=warning msg="Could not add /dev/mshv to the devices cgroup" name=containerd-shim-v2 pid=186134 sandbox=3a2de150104295fa6cbec94e50548807017821fb8cdb79ea34ab9b5dd90ffdb7 source=cgroups
# Jul 12 17:16:01 ubuntu kata[186134]: time="2023-07-12T17:16:01.02743407Z" level=warning msg="Confidential guests do not support hotplugging of vCPUs. Setting DefaultMaxVCPUs to NumVCPUs (1)" name=containerd-shim-v2 pid=186134 sandbox=3a2de150104295fa6cbec94e50548807017821fb8cdb79ea34ab9b5dd90ffdb7 source=virtcontainers/hypervisor
# Jul 12 17:16:01 ubuntu kata[186134]: time="2023-07-12T17:16:01.027857682Z" level=error msg="Create new sandbox failed" error="This system doesn't support Confidential Computing (Guest Protection)" name=containerd-shim-v2 pid=186134 sandbox=3a2de150104295fa6cbec94e50548807017821fb8cdb79ea34ab9b5dd90ffdb7 source=virtcontainers subsystem=sandbox
# Jul 12 17:16:13 ubuntu kata[186270]: time="2023-07-12T17:16:13.045023946Z" level=info msg="loaded configuration" file=/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-snp.toml format=TOML name=containerd-shim-v2 pid=186270 sandbox=cd1757e1b22d1c59495ca59a4d9fbde0473da749a0e71e7663fb2ed0f0b672f5 source=katautils
# Jul 12 17:16:13 ubuntu kata[186270]: time="2023-07-12T17:16:13.045141643Z" level=info msg="IOMMUPlatform is disabled by default." name=containerd-shim-v2 pid=186270 sandbox=cd1757e1b22d1c59495ca59a4d9fbde0473da749a0e71e7663fb2ed0f0b672f5 source=katautils
# Jul 12 17:16:13 ubuntu kata[186270]: time="2023-07-12T17:16:13.047743101Z" level=info msg="shm-size detected: 67108864" source=virtcontainers subsystem=oci
# Jul 12 17:16:13 ubuntu kata[186270]: time="2023-07-12T17:16:13.048755095Z" level=warning msg="Could not add /dev/mshv to the devices cgroup" name=containerd-shim-v2 pid=186270 sandbox=cd1757e1b22d1c59495ca59a4d9fbde0473da749a0e71e7663fb2ed0f0b672f5 source=cgroups
# Jul 12 17:16:13 ubuntu kata[186270]: time="2023-07-12T17:16:13.076046905Z" level=warning msg="Confidential guests do not support hotplugging of vCPUs. Setting DefaultMaxVCPUs to NumVCPUs (1)" name=containerd-shim-v2 pid=186270 sandbox=cd1757e1b22d1c59495ca59a4d9fbde0473da749a0e71e7663fb2ed0f0b672f5 source=virtcontainers/hypervisor
# Jul 12 17:16:13 ubuntu kata[186270]: time="2023-07-12T17:16:13.076481355Z" level=error msg="Create new sandbox failed" error="This system doesn't support Confidential Computing (Guest Protection)" name=containerd-shim-v2 pid=186270 sandbox=cd1757e1b22d1c59495ca59a4d9fbde0473da749a0e71e7663fb2ed0f0b672f5 source=virtcontainers subsystem=sandbox
# Deleting test services...
# service "snp-unencrypted" deleted
# deployment.apps "snp-unencrypted" deleted
# pod/snp-unencrypted-68dd975f64-qjb27 condition met
# Error response from daemon: network simple-kbs_default not found
# /home/ubuntu/go/src/github.com/kata-containers/tests/integration/kubernetes/confidential/../../confidential/lib.sh: line 410: mysql: command not found
[run_kubernetes_tests.sh:153] ERROR: bats confidential/snp.bats

@wainersm
Copy link
Contributor Author

/test

@wainersm wainersm mentioned this pull request Jul 12, 2023
Merged
@wainersm
Copy link
Contributor Author

Now SNP is failing... :(

15:40:00 # Jul 12 18:39:44 amd-milan-coco-ci-ubuntu2004-001 kata[3449707]: time="2023-07-12T18:39:44.61424691Z" level=info msg="Sandbox is started" name=containerd-shim-v2 pid=3449707 sandbox=2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760 source=virtcontainers subsystem=sandbox
15:40:00 # Jul 12 18:39:45 amd-milan-coco-ci-ubuntu2004-001 kata[3449707]: time="2023-07-12T18:39:45.761781152Z" level=error msg="container create failed" container=b384125fc5ecc9f612f3a454329dc052183a308e1090bcd38805fd960e568104 error="no such file or directory" name=containerd-shim-v2 pid=3449707 sandbox=2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760 source=virtcontainers subsystem=container
15:40:00 # Jul 12 18:39:45 amd-milan-coco-ci-ubuntu2004-001 kata[3449707]: time="2023-07-12T18:39:45.761908208Z" level=warning error="no such file or directory" name=containerd-shim-v2 pid=3449707 sandbox=2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760 share-dir=/run/kata-containers/shared/sandboxes/2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760/mounts/b384125fc5ecc9f612f3a454329dc052183a308e1090bcd38805fd960e568104/rootfs source=virtcontainers subsystem=mount
15:40:00 # Jul 12 18:39:45 amd-milan-coco-ci-ubuntu2004-001 kata[3449707]: time="2023-07-12T18:39:45.761965836Z" level=warning msg="Could not remove container share dir" error="no such file or directory" name=containerd-shim-v2 pid=3449707 sandbox=2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760 share-dir=/run/kata-containers/shared/sandboxes/2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760/mounts/b384125fc5ecc9f612f3a454329dc052183a308e1090bcd38805fd960e568104 source=virtcontainers subsystem=fs_share
15:40:00 # Jul 12 18:39:47 amd-milan-coco-ci-ubuntu2004-001 kata[3449707]: time="2023-07-12T18:39:47.260620545Z" level=error msg="container create failed" container=ad4f754869f10d5dda36aea9d26c53d280a7ac68b51feef3e7c8ff1de0c7c545 error="no such file or directory" name=containerd-shim-v2 pid=3449707 sandbox=2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760 source=virtcontainers subsystem=container
15:40:00 # Jul 12 18:39:47 amd-milan-coco-ci-ubuntu2004-001 kata[3449707]: time="2023-07-12T18:39:47.260749725Z" level=warning error="no such file or directory" name=containerd-shim-v2 pid=3449707 sandbox=2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760 share-dir=/run/kata-containers/shared/sandboxes/2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760/mounts/ad4f754869f10d5dda36aea9d26c53d280a7ac68b51feef3e7c8ff1de0c7c545/rootfs source=virtcontainers subsystem=mount
15:40:00 # Jul 12 18:39:47 amd-milan-coco-ci-ubuntu2004-001 kata[3449707]: time="2023-07-12T18:39:47.260824024Z" level=warning msg="Could not remove container share dir" error="no such file or directory" name=containerd-shim-v2 pid=3449707 sandbox=2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760 share-dir=/run/kata-containers/shared/sandboxes/2b55aa460fb972586083c13c2dc870626712655f19f2d89ff569275e4da97760/mounts/ad4f754869f10d5dda36aea9d26c53d280a7ac68b51feef3e7c8ff1de0c7c545 source=virtcontainers subsystem=fs_share

@wainersm wainersm closed this Jul 12, 2023
@wainersm wainersm deleted the tests_amd_improvements branch July 12, 2023 19:39
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@stevenhorsman stevenhorsman stevenhorsman approved these changes

Assignees
No one assigned
Labels
area/testing size/small Small and simple task
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wainersm @stevenhorsman @katacontainersbot