From 03ef7782733a3d0337a2f5908f06e4674bebbec1 Mon Sep 17 00:00:00 2001 From: nadav mizrahi Date: Tue, 27 Jun 2023 09:56:18 +0300 Subject: [PATCH] more code review fixes Signed-off-by: nadav mizrahi --- .../ceph_s3_tests/ceph_s3_tests_guide.md | 58 +++++++++++-------- .../ceph_s3_tests_pending_list_status.md | 3 +- .../s3-tests-lists/s3_tests_black_list.txt | 39 +------------ .../s3-tests-lists/s3_tests_pending_list.txt | 37 ++++++++++++ 4 files changed, 74 insertions(+), 63 deletions(-) diff --git a/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_guide.md b/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_guide.md index 71f1abc5e1..f6d9e1b876 100644 --- a/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_guide.md +++ b/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_guide.md @@ -8,10 +8,10 @@ 4) Debug a Single Ceph S3 Test 5) Compare to AWS Response (Inside Tester Pod) 6) Examples -* This guide describes developer steps to run Ceph S3 on a noobaa system on minikube. +* This guide describes developer steps to run Ceph S3 on a Noobaa system on minikube. ## General Settings For Ceph S3 Tests -We assume that it is not your first deployment of noobaa system, and you already succeeded with it (If not, please see the guide Deploy Noobaa On Minikube). +We assume that it is not your first deployment of Noobaa system, and you already succeeded with it (If not, please see the guide Deploy Noobaa On Minikube). We will run the commands in the terminal, you may work with at least two tabs: 1) For noobaa-core repository 2) For noobaa-operator repository @@ -47,14 +47,14 @@ docker tag noobaa:latest noobaa-core:s3-tests ### 4) Deploy Noobaa (Noobaa-Operator Tab) ```bash nb install --mini --noobaa-image='noobaa-core:s3-tests' -# or use dev flag for higher resources +# or use the dev flag for higher resources nb install --dev --noobaa-image='noobaa-core:s3-tests' ``` _Note: We have the alias to `nb` from the step 'Build Operator'._ The installation should take 5-10 minutes. -Once noobaa is installed please notice that the phase is Ready, you will see it in the CLI logs: +Once Noobaa is installed please notice that the phase is Ready, you will see it in the CLI logs: ✅ System Phase is "Ready". @@ -66,7 +66,7 @@ noobaa-core-0 1/1 Running 0 noobaa-db-pg-0 1/1 Running 0 51m noobaa-default-backing-store-noobaa-pod-a586c55b 1/1 Running 0 47m noobaa-endpoint-6cf5cccfc6-rmdrd 1/1 Running 0 47m -noobaa-operator-5c959d5564-qzgqb 1/1 Running 0 51m +noobaa-operator-5c959d5564-qzgqb 2/2 Running 0 51m ``` ### 5) Wait For Default Backingstore to Be Ready (Noobaa-Operator Tab) @@ -75,6 +75,8 @@ We will use the default backingstore pod to run the tests, we need it to be in p kubectl wait --for=condition=available backingstore/noobaa-default-backing-store --timeout=6m ``` +Note that the default backing store might not be up as soon as the noobaa installation completes. For this reason it is advised to run `kubectl get pods` to make sure the default backing store is up. In case its not, wait for it to be up. If you run kubectl wait on the backing store before its up, the command will fail. + ## Run All Ceph S3 Tests ### 1) Prerequisites: @@ -93,14 +95,15 @@ kubectl logs job/noobaa-tests-s3 -f We run all the tests except the tests that appear in the lists `src/test/system_tests/ceph_s3_tests/s3-tests-lists` if you would like to add or remove a test you can edit those files (and then repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above). -Some tests are marked to be skipped in the tests code. usually because the enviorment doesn't support certain criteria. those tests will appear with a `[WARN]` tag and will be marked as "Test Skipped", for example: +Some tests are marked to be skipped in the code of ceph/s3-tests repository. Usually, because the environment doesn't support certain criteria. Those tests will appear with a `[WARN]` tag and will be marked as "Test Skipped", for example: `[WARN] CONSOLE:: Test skipped: s3tests_boto3/functional/test_s3.py::test_lifecycle_transition` In the test code the function: `pytest.skip("")` will mark them to be skipped. #### Boto3 tests -As mentioned in s3-tests repo (https://github.com/ceph/s3-tests#readme): +As mentioned in the [readme of ceph/s3-tests](https://github.com/ceph/s3-tests#readme): + Most of the tests have both Boto3 and Boto2 versions. Tests written in Boto2 are in the s3tests directory. Tests written in Boto3 are located in the s3test_boto3 directory. In order to only run the boto3 version, tests in the boto directory run with `-- -m 'not fails_on_rgw`: @@ -115,13 +118,13 @@ S3TEST_CONF=your.conf tox -- -m 'not fails_on_rgw' s3tests_boto3/functional::tes Following the 'General Settings For Ceph S3 Tests' steps. ### 2) Increasing Debug Level (Noobaa-Operator) -Before running a test, you can increase the debug level with noobaa CLI. +Before running a test, you can increase the debug level with Noobaa CLI. ```bash nb system set-debug-level 1 ``` A good level to start with is 1, the higher you go the more verbose and noisy the logs will become (it is recommended using 3 level at the most for those tests). -Tip: If there is an existing printing in higher level than 1 and you only want to see it (or you wish to add a certain printing) change the debug level of the printing in the code to 0 (repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above), for example: +Tip: If there is an existing printing in a higher level than 1 and you only want to see it (or you wish to add a certain printing) change the debug level of the printing in the code to 0 (repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above), for example: ```diff - dbg.log2('message'); @@ -160,6 +163,12 @@ node ./src/test/system_tests/ceph_s3_tests/test_ceph_s3_config_setup.js If you want to disable summary warnings add the following flag to the test command: `-- --disable-pytest-warnings` +For example: + +```bash +S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf tox -c src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini -- --disable-pytest-warnings ${PWD}/src/test/system_tests/ceph_s3_tests/s3-tests/s3tests/functional/test_headers.py::test_bucket_create_contentlength_none +``` + Note that every flag that comes after `--` is passed to pytest from tox. so if there is already `--` in the command just put `--disable-pytest-warnings` as part of the flags after it, no need to add another `--` notation. for example to add --disable-pytest-warnings to the command: @@ -180,24 +189,22 @@ S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf This should run the test on the noobaa deployment we've set up. #### Test Name -You can find a list of tests in the doc inside the file `ceph_s3_tests_list_single_test.txt`. Please notice that the test name has a certain structure: directories are separated with `/`, the files end with the extention `.py` and the function to run (usually with a prefix `test_`) appears after the `::` sign. +You can find a list of tests in the doc inside the file `ceph_s3_tests_list_single_test.txt`. Please notice that the test name has a certain structure: directories are separated with `/`, the files end with the extension `.py` and the function to run (usually with a prefix `test_`) appears after the `::` sign. -in case the test name is incorrect, for example if you add `:` instead of `::` to the test name, the command will fail. -the error will be `file or directory not found` and pytest will exit with error code 4 (which means "pytest command line usage error") +In case the test name is incorrect, for example if you add `:` instead of `::` to the test name, the command will fail. +The error will be `file or directory not found` and pytest will exit with error code 4 (which means "pytest command line usage error") ## Debug a Single Test (Inside The Tester Pod) ### 1) Prerequisites: Following the 'Run a Single Ceph S3 Test' steps. ### 2) View The Test Content -You can view the test by going to the test file and searching for the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vi ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`. +You can view the test by going to the test file and searching for the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vim ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`. The best place to start investigating is noobaa endpoint pod logs. if you are running with debug level that is higher than 1, you should see log messages of the S3 requests with the prefix `S3 REQUEST`. S3 replies will be with the prefix `HTTP REPLY`. ### 3) Change a Test Sometimes you would like to change a test: add printing of variables, skip an assertion as needed, or you suspect that it has a faulty and you would like to change the code. -Since the file `./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` is a read-only file, decide on one of the options: - #### A. Permanent change - this change will be saved in a repo, it is for continues investigating. 1) Fork and clone the repository [ceph/s3-test](https://github.com/ceph/s3-tests). 2) Create a new branch from the hash number that was set in the file `./src/test/system_tests/ceph_s3_tests/test_ceph_s3_deploy.sh`. @@ -206,7 +213,7 @@ Since the file `./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/fun 5) Build the tester image again, deploy noobaa, and run the test (repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above). #### B. Temporary change - this change will be saved in the file inside the container, useful when you need a small change. -You can edit a test by going to the test file and editing the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vi ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`. +You can edit a test by going to the test file and editing the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vim ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`. ## Compare to AWS Response (Inside Tester Pod) Prerequisites: Following the 'Run a Single Ceph S3 Test' steps until 'Deploy The Tester Deployment (Noobaa-Core Tab)'. @@ -221,7 +228,6 @@ In this section we will do some manual changes that will allow you to check AWS vim src/test/system_tests/ceph_s3_tests/test_ceph_s3_config_aws.conf ``` * host = s3.amazonaws.com -* bucket prefix = choose_name (for example: `bucket prefix = foo-bucket` you will need to manually delete it from AWS, and its name will be `foo-bucket1`, it adds suffix of 1). * access_key, secret_key appears 3 times each in the file. 3) Running tests with the new configuration files will run against AWS: ```bash @@ -235,15 +241,19 @@ S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config_aws.c Following the 'Run All Ceph S3 Tests' steps. ### 1) All Running Tested Passed -a snippet from the last part of running all the tests. +a snippet from the last part of running all the tests, Before the list of skipped and failed tests. You can see how many tests run and a status for each test. ``` ... -Test Passed: s3tests_boto3.functional.test_sts.test_assume_role_with_web_identity_resource_tag_copy_obj -Test Passed: s3tests_boto3.functional.test_sts.test_assume_role_with_web_identity_role_resource_tag -Test Passed: s3tests_boto3.functional.test_utils.test_generate -Finished Running Ceph S3 Tests -CEPH TEST SUMMARY: Suite contains 812, ran 387 tests, Passed: 387, Skipped: 0, Failed: 0 +Test Passed: s3tests_boto3/functional/test_s3select.py::test_bool_cast_expressions +Test Passed: s3tests_boto3/functional/test_s3select.py::test_output_serial_expressions +Test Passed: s3tests_boto3/functional/test_utils.py::test_generate +inished Running Ceph S3 Tests +CEPH TEST SUMMARY: Suite contains 779, ran 336 tests, Passed: 310, Skipped: 26, Failed: 0 +CEPH TEST SKIPPED TESTS SUMMARY: 26 skipped tests +s3tests/functional/test_s3.py::test_object_storage_class +s3tests/functional/test_s3.py::test_object_storage_class_multipart +... ``` ## Running a Single Test @@ -259,7 +269,7 @@ S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf Note that there is the warning: `WARNING: could not copy distfile to //.tox/distshare` -this warning is for tox to use the same dependancies between projects. this feature is depricated and not used on this project. in order to remove the warning you can modify src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini to include the following line in the `[tox]` section: `distshare = /root/node_modules/noobaa-core/.tox/distshare` +this warning is for tox to use the same dependancies between projects. this feature is deprecated and not used on this project. In order to remove the warning you can modify src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini to include the following line in the `[tox]` section: `distshare = /root/node_modules/noobaa-core/.tox/distshare` ### 2) Test Fail For example: diff --git a/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_pending_list_status.md b/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_pending_list_status.md index 24466ffb15..03d5a4cc86 100644 --- a/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_pending_list_status.md +++ b/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_pending_list_status.md @@ -36,4 +36,5 @@ Attached a table with tests that where investigated and their status (this table | test_schema_definition | Faulty Test | | Same as test_alias_cyclic_refernce | | test_progress_expressions | Faulty Test | [508](https://github.com/ceph/s3-tests/issues/508) | | | test_lifecycle_set_date | Faulty Test | [510](https://github.com/ceph/s3-tests/issues/510) | | -| test_lifecycle_transition_set_invalid_date | Not Implemented | | added because of the following PR [7270](https://github.com/noobaa/noobaa-core/pull/7270#discussion_r1175123422) | \ No newline at end of file +| test_lifecycle_transition_set_invalid_date | Not Implemented | | added because of the following PR [7270](https://github.com/noobaa/noobaa-core/pull/7270#discussion_r1175123422) | +| All aws4 tests (e.g test_bucket_create_bad_amz_date_before_epoch_aws4) | Faulty Test | [520](https://github.com/ceph/s3-tests/issues/520) | | \ No newline at end of file diff --git a/src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_black_list.txt b/src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_black_list.txt index 59c9b78206..849f5a712d 100644 --- a/src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_black_list.txt +++ b/src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_black_list.txt @@ -352,41 +352,4 @@ s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_resourc s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_wrong_resource_tag_deny s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_resource_tag_princ_tag s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_resource_tag_copy_obj -s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_role_resource_tag -s3tests/functional/test_headers.py::test_object_create_bad_md5_invalid_garbage_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_contentlength_mismatch_below_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_authorization_incorrect_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_authorization_invalid_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_ua_empty_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_ua_none_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_date_invalid_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_amz_date_invalid_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_date_empty_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_amz_date_empty_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_date_none_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_amz_date_none_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_date_before_today_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_amz_date_before_today_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_date_after_today_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_amz_date_after_today_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_date_before_epoch_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_amz_date_before_epoch_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_date_after_end_aws4 -s3tests/functional/test_headers.py::test_object_create_bad_amz_date_after_end_aws4 -s3tests/functional/test_headers.py::test_object_create_missing_signed_custom_header_aws4 -s3tests/functional/test_headers.py::test_object_create_missing_signed_header_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_invalid_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_ua_empty_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_ua_none_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_date_invalid_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_invalid_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_date_empty_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_empty_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_date_none_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_none_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_today_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_before_today_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_date_after_today_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_after_today_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_epoch_aws4 -s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_before_epoch_aws4 \ No newline at end of file +s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_role_resource_tag \ No newline at end of file diff --git a/src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_pending_list.txt b/src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_pending_list.txt index a06d24d63c..eb5305c681 100644 --- a/src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_pending_list.txt +++ b/src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_pending_list.txt @@ -77,3 +77,40 @@ s3tests_boto3/functional/test_s3.py::test_get_object_torrent s3tests_boto3/functional/test_s3select.py::test_count_json_operation s3tests_boto3/functional/test_s3select.py::test_column_sum_min_max s3tests_boto3/functional/test_s3.py::test_multipart_upload_resend_part +s3tests/functional/test_headers.py::test_object_create_bad_md5_invalid_garbage_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_contentlength_mismatch_below_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_authorization_incorrect_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_authorization_invalid_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_ua_empty_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_ua_none_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_date_invalid_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_amz_date_invalid_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_date_empty_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_amz_date_empty_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_date_none_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_amz_date_none_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_date_before_today_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_amz_date_before_today_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_date_after_today_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_amz_date_after_today_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_date_before_epoch_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_amz_date_before_epoch_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_date_after_end_aws4 +s3tests/functional/test_headers.py::test_object_create_bad_amz_date_after_end_aws4 +s3tests/functional/test_headers.py::test_object_create_missing_signed_custom_header_aws4 +s3tests/functional/test_headers.py::test_object_create_missing_signed_header_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_invalid_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_ua_empty_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_ua_none_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_date_invalid_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_invalid_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_date_empty_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_empty_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_date_none_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_none_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_today_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_before_today_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_date_after_today_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_after_today_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_epoch_aws4 +s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_before_epoch_aws4