diff --git a/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_guide.md b/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_guide.md index b4d957bf68..9b0688a7c1 100644 --- a/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_guide.md +++ b/docs/dev_guide/ceph_s3_tests/ceph_s3_tests_guide.md @@ -89,6 +89,8 @@ kubectl apply -f src/test/system_tests/ceph_s3_tests/test_ceph_s3_job.yml kubectl logs job/noobaa-tests-s3 -f ``` +#### Skipped tests + We run all the tests except the tests that appear in the lists `src/test/system_tests/ceph_s3_tests/s3-tests-lists` if you would like to add or remove a test you can edit those files (and then repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above). Some tests are marked to be skipped in the tests code. usually because the enviorment doesn't support certain criteria. those tests will appear with a `[WARN]` tag and will be marked as "Test Skipped", for example: @@ -97,6 +99,13 @@ Some tests are marked to be skipped in the tests code. usually because the envio In the test code the function: `pytest.skip("")` will mark them to be skipped. +#### Boto3 tests +Most of the tests have both Boto3 and Boto2 versions. Tests written in Boto2 are in the s3tests directory. Tests written in Boto3 are located in the s3test_boto3 directory. + +in order to only run the boto3 version, tests in the boto directory run with `-- -m 'not fails_on_aws`: + +`S3TEST_CONF=your.conf tox -- -m 'not fails_on_aws' s3tests_boto3/functional::test_name` + ## Run a Single Ceph S3 Test ### 1) Prerequisites: @@ -144,10 +153,10 @@ Run the script that will create the necessary accounts in noobaa and update the node ./src/test/system_tests/ceph_s3_tests/test_ceph_s3_config_setup.js ``` -Note: If you want to ignore PythonDeprecationWarnings add the following flag to the test command (which will then ignore all Python warnings, so keep that in mind): +If you want to summary warnings add the following flag to the test command: `-- --disable-pytest-warnings` -note that every flag that comes after `--` is past to pytest from tox. so if there is already `--` in the command just put `--disable-pytest-warnings` as part of the flags after it, no need to add another `--` notation. +Note that every flag that comes after `--` is passed to pytest from tox. so if there is already `--` in the command just put `--disable-pytest-warnings` as part of the flags after it, no need to add another `--` notation. for example to add --disable-pytest-warnings to the command: `S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf tox -c src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini -- -m 'not fails_on_aws' ${PWD}/src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py::test_account_usage` @@ -163,7 +172,7 @@ S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf This should run the test on the noobaa deployment we've set up. #### Test Name -You can find a list of tests in the doc inside the file `ceph_s3_tests_list_single_test.txt`. Please notice that the test name has a certain structure are separated with `/`, the files end with the extention `.py` and the function to run (usually with a prefix `test_`) appears after the `::` sign. +You can find a list of tests in the doc inside the file `ceph_s3_tests_list_single_test.txt`. Please notice that the test name has a certain structure: directories are separated with `/`, the files end with the extention `.py` and the function to run (usually with a prefix `test_`) appears after the `::` sign. ## Debug a Single Test (Inside The Tester Pod) ### 1) Prerequisites: @@ -186,13 +195,13 @@ Since the file `./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/fun 5) Build the tester image again, deploy noobaa, and run the test (repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above). #### B. Temporary change - this change will be saved in the file inside the container, useful when you need a small change. -You can edit the test by going to the test file and editing the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vi ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`. -### Compare to AWS Response (Inside Tester Pod) +You can edit a test by going to the test file and editing the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vi ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`. +## Compare to AWS Response (Inside Tester Pod) Prerequisites: Following the 'Run a Single Ceph S3 Test' steps until 'Deploy The Tester Deployment (Noobaa-Core Tab)'. In this section we will do some manual changes that will allow you to check AWS response for a specific test (tests that do not use neither ACL nor tenant group). -1) copy configuration file: +1) Copy configuration file - this will allow us to run a test on AWS and then back to NooBaa just by changing the configuration file (we would have 2 configuration files: `test_ceph_s3_config.conf` and `test_ceph_s3_config_aws.conf`): ```bash cp src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf src/test/system_tests/ceph_s3_tests/test_ceph_s3_config_aws.conf ``` diff --git a/src/test/system_tests/ceph_s3_tests/test_ceph_s3.js b/src/test/system_tests/ceph_s3_tests/test_ceph_s3.js index f48c974d8a..787658de0f 100644 --- a/src/test/system_tests/ceph_s3_tests/test_ceph_s3.js +++ b/src/test/system_tests/ceph_s3_tests/test_ceph_s3.js @@ -99,34 +99,35 @@ async function test_worker() { } } -async function run_single_test(test_name) { +async function run_single_test(test) { let ceph_args = `S3TEST_CONF=${process.cwd()}/${CEPH_TEST.test_dir}${CEPH_TEST.ceph_config}`; - if (S3_CEPH_TEST_SIGV4.includes(test_name)) { + if (S3_CEPH_TEST_SIGV4.includes(test)) { ceph_args += ` S3_USE_SIGV4=true`; } let base_cmd = `${ceph_args} tox ${TOX_ARGS}`; - if (!S3_CEPH_TEST_OUT_OF_SCOPE_REGEXP.test(test_name)) { + if (!S3_CEPH_TEST_OUT_OF_SCOPE_REGEXP.test(test)) { try { - if (test_name.includes('boto')) { + if (test.includes('boto')) { base_cmd = `${ceph_args} tox ${TOX_ARGS} -- -m 'not fails_on_aws'`; } - const res = await os_utils.exec(`${base_cmd} ${process.cwd()}/${CEPH_TEST.test_dir}${CEPH_TEST.s3_test_dir}${test_name}`, { ignore_rc: false, return_stdout: true }); - if (res.indexOf('skipped') >= 0) { - console.warn('Test skipped:', test_name); - testing_status.skip.push(test_name); + const full_test_command = `${base_cmd} ${process.cwd()}/${CEPH_TEST.test_dir}${CEPH_TEST.s3_test_dir}${test}`; + const res = await os_utils.exec(full_test_command, { ignore_rc: false, return_stdout: true }); + if (res.includes('skipped')) { + console.warn('Test skipped:', test); + testing_status.skip.push(test); } else { - console.info('Test Passed:', test_name); - testing_status.pass.push(test_name); + console.info('Test Passed:', test); + testing_status.pass.push(test); } } catch (err) { // tox will exit with code 1 on error regardless of pytest exit code. pytest exit code 5 means no tests ran. // can happen when 'not fails_on_aws' flag is on for some tests (there are no boto3 tests for the test) - if (err.stdout.indexOf("exited with code 5") >= 0) { - console.warn('Test skipped:', test_name); - testing_status.skip.push(test_name); + if (err.stdout.includes("exited with code 5")) { + console.warn('Test skipped:', test); + testing_status.skip.push(test); } else { - console.error('Test Failed:', test_name); - testing_status.fail.push(test_name); + console.error('Test Failed:', test); + testing_status.fail.push(test); } } }