more code review fixes

Signed-off-by: nadav mizrahi <[email protected]>

.github/workflows/ceph-s3-tests.yaml

@@ -74,6 +74,7 @@ jobs:
           kubectl logs job/noobaa-tests-s3 --tail 10000 -f
           if kubectl logs job/noobaa-tests-s3 | grep -q "Ceph Test Failed:"; then
             echo "At least one test failed!"
+            kubectl get pods
             exit 1
           fi
           if [ ${TIMEOUT} ]; then

docs/dev_guide/ceph_s3_tests/ceph_s3_tests_guide.md

@@ -89,6 +89,8 @@ kubectl apply -f src/test/system_tests/ceph_s3_tests/test_ceph_s3_job.yml
 kubectl logs job/noobaa-tests-s3 -f
 ```
 
+#### Skipped tests
+
 We run all the tests except the tests that appear in the lists `src/test/system_tests/ceph_s3_tests/s3-tests-lists` if you would like to add or remove a test you can edit those files (and then repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above).
 
 Some tests are marked to be skipped in the tests code. usually because the enviorment doesn't support certain criteria. those tests will appear with a `[WARN]` tag and will be marked as "Test Skipped", for example:
@@ -97,6 +99,14 @@ Some tests are marked to be skipped in the tests code. usually because the envio
 In the test code the function:
 `pytest.skip("")` will mark them to be skipped.
 
+#### Boto3 tests
+As mentioned in s3-tests repo (https://github.com/ceph/s3-tests#readme):
+Most of the tests have both Boto3 and Boto2 versions. Tests written in Boto2 are in the s3tests directory. Tests written in Boto3 are located in the s3test_boto3 directory.
+
+In order to only run the boto3 version, tests in the boto directory run with `-- -m 'not fails_on_rgw`:
+
+`S3TEST_CONF=your.conf tox -- -m 'not fails_on_rgw' s3tests_boto3/functional::test_name`
+
 ## Run a Single Ceph S3 Test
 
 ### 1) Prerequisites:
@@ -144,16 +154,17 @@ Run the script that will create the necessary accounts in noobaa and update the
 node ./src/test/system_tests/ceph_s3_tests/test_ceph_s3_config_setup.js
 ```
 
-Note: If you want to ignore PythonDeprecationWarnings add the following flag to the test command (which will then ignore all Python warnings, so keep that in mind):
+#### Disable pytest warnings
+If you want to disable summary warnings add the following flag to the test command:
 `-- --disable-pytest-warnings`
 
-note that every flag that comes after `--` is past to pytest from tox. so if there is already `--` in the command just put `--disable-pytest-warnings` as part of the flags after it, no need to add another `--` notation.
+Note that every flag that comes after `--` is passed to pytest from tox. so if there is already `--` in the command just put `--disable-pytest-warnings` as part of the flags after it, no need to add another `--` notation.
 
 for example to add --disable-pytest-warnings to the command:
-`S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf tox -c src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini -- -m 'not fails_on_aws' ${PWD}/src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py::test_account_usage`
+`S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf tox -c src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini -- -m 'not fails_on_rgw' ${PWD}/src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py::test_account_usage`
 
 it should be:
-`S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf tox -c src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini -- -m 'not fails_on_aws --disable-pytest-warnings ${PWD}/src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py::test_account_usage`
+`S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf tox -c src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini -- -m 'not fails_on_rgw --disable-pytest-warnings ${PWD}/src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py::test_account_usage`
 
 ### 5) Run a Test (Inside The Tester Pod)
 To run a test, from noobaa working directory:
@@ -163,7 +174,10 @@ S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf
 This should run the test on the noobaa deployment we've set up.
 
 #### Test Name
-You can find a list of tests in the doc inside the file `ceph_s3_tests_list_single_test.txt`. Please notice that the test name has a certain structure <directory_name> are separated with `/`, the files end with the extention `.py` and the function to run (usually with a prefix `test_`) appears after the `::` sign.
+You can find a list of tests in the doc inside the file `ceph_s3_tests_list_single_test.txt`. Please notice that the test name has a certain structure: directories are separated with `/`, the files end with the extention `.py` and the function to run (usually with a prefix `test_`) appears after the `::` sign.
+
+in case the test name is incorrect, for example if you add `:` instead of `::` to the test name, the command will fail.
+the error will be `file or directory not found` and pytest will exit with error code 4 (which means "pytest command line usage error")
 ## Debug a Single Test (Inside The Tester Pod)
 
 ### 1) Prerequisites:
@@ -186,13 +200,13 @@ Since the file `./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/fun
 5) Build the tester image again, deploy noobaa, and run the test (repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above).
 
 #### B. Temporary change - this change will be saved in the file inside the container, useful when you need a small change.
-You can edit the test by going to the test file and editing the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vi ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`.
-### Compare to AWS Response (Inside Tester Pod)
+You can edit a test by going to the test file and editing the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vi ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`.
+## Compare to AWS Response (Inside Tester Pod)
 Prerequisites:
 Following the 'Run a Single Ceph S3 Test' steps until 'Deploy The Tester Deployment (Noobaa-Core Tab)'.
 
 In this section we will do some manual changes that will allow you to check AWS response for a specific test (tests that do not use neither ACL nor tenant group).
-1) copy configuration file:
+1) Copy configuration file - this will allow us to run a test on AWS and then back to NooBaa just by changing the configuration file (we would have 2 configuration files: `test_ceph_s3_config.conf` and `test_ceph_s3_config_aws.conf`):
   ```bash
   cp src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf src/test/system_tests/ceph_s3_tests/test_ceph_s3_config_aws.conf
   ```
@@ -228,13 +242,29 @@ CEPH TEST SUMMARY: Suite contains 812, ran 387 tests, Passed: 387, Skipped: 0, F
 Following the 'Run a Single Ceph S3 Test' steps.
 
 ### 1) Test Pass
-For example: `s3tests_boto3/functional/test_s3.py::test_basic_key_count`
-![alt text](images/tox_test_success.png)
+For example: 
+```bash
+S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf tox -c src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini -- --disable-pytest-warnings ${PWD}/src/test/system_tests/ceph_s3_tests/s3-tests/s3tests/functional/test_headers.py::test_bucket_create_contentlength_none
+```
+![test pass screenshot](images/tox_test_pass.png)
 
-note that there is the warning:
+Note that there is the warning:
 `WARNING: could not copy distfile to //.tox/distshare`
 this warning is for tox to use the same dependancies between projects. this feature is depricated and not used on this project. in order to remove the warning you can modify src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini to include the following line in the `[tox]` section: `distshare = /root/node_modules/noobaa-core/.tox/distshare`
 
 ### 2) Test Fail
-For example: `s3tests_boto3/functional/test_s3::test_account_usage.py`
-![alt text](images/tox_test_failed.png)
+For example: 
+```bash
+S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf tox -c src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini -- --disable-pytest-warnings ${PWD}/src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py::test_account_usage
+```
+![test failed screenshot](images/tox_test_failed.png)
+
+### 3) Test Skipped
+```bash
+S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf tox -c src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini -- --disable-pytest-warnings ${PWD}/src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py::test_bucket_get_location
+```
+![test skipped screenshot](images/tox_test_skipped.png)
+
+notice that even though test commands succeeded the test itself was skipped. the test prints `1 skipped` meaning one test was skipped
+
+

src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_black_list.txt

@@ -352,4 +352,41 @@ s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_resourc
 s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_wrong_resource_tag_deny
 s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_resource_tag_princ_tag
 s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_resource_tag_copy_obj
-s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_role_resource_tag
+s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_role_resource_tag
+s3tests/functional/test_headers.py::test_object_create_bad_md5_invalid_garbage_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_contentlength_mismatch_below_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_authorization_incorrect_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_authorization_invalid_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_ua_empty_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_ua_none_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_date_invalid_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_amz_date_invalid_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_date_empty_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_amz_date_empty_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_date_none_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_amz_date_none_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_date_before_today_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_amz_date_before_today_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_date_after_today_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_amz_date_after_today_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_date_before_epoch_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_amz_date_before_epoch_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_date_after_end_aws4
+s3tests/functional/test_headers.py::test_object_create_bad_amz_date_after_end_aws4
+s3tests/functional/test_headers.py::test_object_create_missing_signed_custom_header_aws4
+s3tests/functional/test_headers.py::test_object_create_missing_signed_header_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_invalid_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_ua_empty_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_ua_none_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_date_invalid_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_invalid_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_date_empty_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_empty_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_date_none_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_none_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_today_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_before_today_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_date_after_today_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_after_today_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_epoch_aws4
+s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_before_epoch_aws4

src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_pending_list.txt

@@ -76,3 +76,4 @@ s3tests_boto3/functional/test_s3.py::test_post_object_upload_size_rgw_chunk_size
 s3tests_boto3/functional/test_s3.py::test_get_object_torrent
 s3tests_boto3/functional/test_s3select.py::test_count_json_operation
 s3tests_boto3/functional/test_s3select.py::test_column_sum_min_max
+s3tests_boto3/functional/test_s3.py::test_multipart_upload_resend_part

src/test/system_tests/ceph_s3_tests/test_ceph_s3.js

@@ -15,7 +15,7 @@ const dbg = require('../../../util/debug_module')(__filename);
 dbg.set_process_name('test_ceph_s3');
 const argv = require('minimist')(process.argv.slice(2));
 delete argv._;
-const { S3_CEPH_TEST_SIGV4, CEPH_TEST, DEFAULT_NUMBER_OF_WORKERS, TOX_ARGS } = require('./test_ceph_s3_constants.js');
+const { CEPH_TEST, DEFAULT_NUMBER_OF_WORKERS, TOX_ARGS, AWS4_TEST_ENDING } = require('./test_ceph_s3_constants.js');
 
 const testing_status = {
     pass: [],
@@ -65,7 +65,7 @@ async function run_s3_tests() {
 
     console.info(`CEPH TEST SUMMARY: Suite contains ${testing_status.total}, ran ${testing_status.pass.length + testing_status.fail.length + testing_status.skip.length} tests, Passed: ${testing_status.pass.length}, Skipped: ${testing_status.skip.length}, Failed: ${testing_status.fail.length}`);
     if (testing_status.skip.length) {
-        console.warn(`CEPH TEST SUMMARY:  ${testing_status.skip.length} skipped tests \n${testing_status.skip.join('\n')}`);
+        console.warn(`CEPH TEST SKIPPED TESTS SUMMARY:  ${testing_status.skip.length} skipped tests \n${testing_status.skip.join('\n')}`);
     }
     if (testing_status.fail.length) {
         console.error(`CEPH TEST FAILED TESTS SUMMARY: ${testing_status.fail.length} failed tests \n${testing_status.fail.join('\n')}`);
@@ -99,34 +99,35 @@ async function test_worker() {
     }
 }
 
-async function run_single_test(test_name) {
+async function run_single_test(test) {
     let ceph_args = `S3TEST_CONF=${process.cwd()}/${CEPH_TEST.test_dir}${CEPH_TEST.ceph_config}`;
-    if (S3_CEPH_TEST_SIGV4.includes(test_name)) {
+    if (test.endsWith(AWS4_TEST_ENDING)) {
         ceph_args += ` S3_USE_SIGV4=true`;
     }
     let base_cmd = `${ceph_args} tox ${TOX_ARGS}`;
-    if (!S3_CEPH_TEST_OUT_OF_SCOPE_REGEXP.test(test_name)) {
+    if (!S3_CEPH_TEST_OUT_OF_SCOPE_REGEXP.test(test)) {
         try {
-            if (test_name.includes('boto')) {
-                base_cmd = `${ceph_args} tox ${TOX_ARGS} -- -m 'not fails_on_aws'`;
+            if (test.includes('boto')) {
+                base_cmd = `${ceph_args} tox ${TOX_ARGS} -- -m 'not fails_on_rgw'`;
             }
-            const res = await os_utils.exec(`${base_cmd} ${process.cwd()}/${CEPH_TEST.test_dir}${CEPH_TEST.s3_test_dir}${test_name}`, { ignore_rc: false, return_stdout: true });
-            if (res.indexOf('skipped') >= 0) {
-                console.warn('Test skipped:', test_name);
-                testing_status.skip.push(test_name);
+            const full_test_command = `${base_cmd} ${process.cwd()}/${CEPH_TEST.test_dir}${CEPH_TEST.s3_test_dir}${test}`;
+            const res = await os_utils.exec(full_test_command, { ignore_rc: false, return_stdout: true });
+            if (res.includes('skipped')) {
+                console.warn('Test skipped:', test);
+                testing_status.skip.push(test);
             } else {
-                console.info('Test Passed:', test_name);
-                testing_status.pass.push(test_name);
+                console.info('Test Passed:', test);
+                testing_status.pass.push(test);
             }
         } catch (err) {
             // tox will exit with code 1 on error regardless of pytest exit code. pytest exit code 5 means no tests ran.
-            // can happen when 'not fails_on_aws' flag is on for some tests (there are no boto3 tests for the test)
-            if (err.stdout.indexOf("exited with code 5") >= 0) {
-                console.warn('Test skipped:', test_name);
-                testing_status.skip.push(test_name);
+            // can happen when 'not fails_on_rgw' flag is on for some tests (there are no boto3 tests for the test)
+            if (err.stdout.includes("exited with code 5")) {
+                console.warn('Test skipped:', test);
+                testing_status.skip.push(test);
             } else {
-                console.error('Test Failed:', test_name);
-                testing_status.fail.push(test_name);
+                console.error('Test Failed:', test);
+                testing_status.fail.push(test);
             }
         }
     }