Skip to content

Commit

Permalink
more code review fixes
Browse files Browse the repository at this point in the history 
Signed-off-by: nadav mizrahi <[email protected]>
  • Loading branch information
@nadavMiz
nadavMiz committed Jun 27, 2023
1 parent 825de58 commit b8e288a
Show file tree
Hide file tree
Showing 4 changed files with 74 additions and 63 deletions.
58 changes: 34 additions & 24 deletions docs/dev_guide/ceph_s3_tests/ceph_s3_tests_guide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@
4) Debug a Single Ceph S3 Test
5) Compare to AWS Response (Inside Tester Pod)
6) Examples
* This guide describes developer steps to run Ceph S3 on a noobaa system on minikube.
* This guide describes developer steps to run Ceph S3 on a Noobaa system on minikube.

## General Settings For Ceph S3 Tests
We assume that it is not your first deployment of noobaa system, and you already succeeded with it (If not, please see the guide Deploy Noobaa On Minikube).
We assume that it is not your first deployment of Noobaa system, and you already succeeded with it (If not, please see the guide Deploy Noobaa On Minikube).
We will run the commands in the terminal, you may work with at least two tabs:
1) For noobaa-core repository
2) For noobaa-operator repository
Expand Down Expand Up @@ -47,14 +47,14 @@ docker tag noobaa:latest noobaa-core:s3-tests
### 4) Deploy Noobaa (Noobaa-Operator Tab)
```bash
nb install --mini --noobaa-image='noobaa-core:s3-tests'
# or use dev flag for higher resources
# or use the dev flag for higher resources
nb install --dev --noobaa-image='noobaa-core:s3-tests'

```
_Note: We have the alias to `nb` from the step 'Build Operator'._

The installation should take 5-10 minutes.
Once noobaa is installed please notice that the phase is Ready, you will see it in the CLI logs:
Once Noobaa is installed please notice that the phase is Ready, you will see it in the CLI logs:

✅ System Phase is "Ready".

Expand All @@ -66,7 +66,7 @@ noobaa-core-0 1/1 Running 0
noobaa-db-pg-0 1/1 Running 0 51m
noobaa-default-backing-store-noobaa-pod-a586c55b 1/1 Running 0 47m
noobaa-endpoint-6cf5cccfc6-rmdrd 1/1 Running 0 47m
noobaa-operator-5c959d5564-qzgqb 1/1 Running 0 51m
noobaa-operator-5c959d5564-qzgqb 2/2 Running 0 51m
```

### 5) Wait For Default Backingstore to Be Ready (Noobaa-Operator Tab)
Expand All @@ -75,6 +75,8 @@ We will use the default backingstore pod to run the tests, we need it to be in p
kubectl wait --for=condition=available backingstore/noobaa-default-backing-store --timeout=6m
```

Note that the default backing store might not be up as soon as the noobaa installation completes. For this reason it is advised to run `kubectl get pods` to make sure the default backing store is up. In case its not, wait for it to be up. If you run kubectl wait on the backing store before its up, the command will fail.

## Run All Ceph S3 Tests

### 1) Prerequisites:
Expand All @@ -93,14 +95,15 @@ kubectl logs job/noobaa-tests-s3 -f

We run all the tests except the tests that appear in the lists `src/test/system_tests/ceph_s3_tests/s3-tests-lists` if you would like to add or remove a test you can edit those files (and then repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above).

Some tests are marked to be skipped in the tests code. usually because the enviorment doesn't support certain criteria. those tests will appear with a `[WARN]` tag and will be marked as "Test Skipped", for example:
Some tests are marked to be skipped in the code of ceph/s3-tests repository. Usually, because the environment doesn't support certain criteria. Those tests will appear with a `[WARN]` tag and will be marked as "Test Skipped", for example:
`[WARN] CONSOLE:: Test skipped: s3tests_boto3/functional/test_s3.py::test_lifecycle_transition`

In the test code the function:
`pytest.skip("")` will mark them to be skipped.

#### Boto3 tests
As mentioned in s3-tests repo (https://github.com/ceph/s3-tests#readme):
As mentioned in the [readme of ceph/s3-tests](https://github.com/ceph/s3-tests#readme):

Most of the tests have both Boto3 and Boto2 versions. Tests written in Boto2 are in the s3tests directory. Tests written in Boto3 are located in the s3test_boto3 directory.

In order to only run the boto3 version, tests in the boto directory run with `-- -m 'not fails_on_rgw`:
Expand All @@ -115,13 +118,13 @@ S3TEST_CONF=your.conf tox -- -m 'not fails_on_rgw' s3tests_boto3/functional::tes
Following the 'General Settings For Ceph S3 Tests' steps.

### 2) Increasing Debug Level (Noobaa-Operator)
Before running a test, you can increase the debug level with noobaa CLI.
Before running a test, you can increase the debug level with Noobaa CLI.
```bash
nb system set-debug-level 1
```
A good level to start with is 1, the higher you go the more verbose and noisy the logs will become (it is recommended using 3 level at the most for those tests).

Tip: If there is an existing printing in higher level than 1 and you only want to see it (or you wish to add a certain printing) change the debug level of the printing in the code to 0 (repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above), for example:
Tip: If there is an existing printing in a higher level than 1 and you only want to see it (or you wish to add a certain printing) change the debug level of the printing in the code to 0 (repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above), for example:

```diff
- dbg.log2('message');
Expand Down Expand Up @@ -160,6 +163,12 @@ node ./src/test/system_tests/ceph_s3_tests/test_ceph_s3_config_setup.js
If you want to disable summary warnings add the following flag to the test command:
`-- --disable-pytest-warnings`

For example:

```bash
S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf tox -c src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini -- --disable-pytest-warnings ${PWD}/src/test/system_tests/ceph_s3_tests/s3-tests/s3tests/functional/test_headers.py::test_bucket_create_contentlength_none
```

Note that every flag that comes after `--` is passed to pytest from tox. so if there is already `--` in the command just put `--disable-pytest-warnings` as part of the flags after it, no need to add another `--` notation.

for example to add --disable-pytest-warnings to the command:
Expand All @@ -180,24 +189,22 @@ S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf
This should run the test on the noobaa deployment we've set up.

#### Test Name
You can find a list of tests in the doc inside the file `ceph_s3_tests_list_single_test.txt`. Please notice that the test name has a certain structure: directories are separated with `/`, the files end with the extention `.py` and the function to run (usually with a prefix `test_`) appears after the `::` sign.
You can find a list of tests in the doc inside the file `ceph_s3_tests_list_single_test.txt`. Please notice that the test name has a certain structure: directories are separated with `/`, the files end with the extension `.py` and the function to run (usually with a prefix `test_`) appears after the `::` sign.

in case the test name is incorrect, for example if you add `:` instead of `::` to the test name, the command will fail.
the error will be `file or directory not found` and pytest will exit with error code 4 (which means "pytest command line usage error")
In case the test name is incorrect, for example if you add `:` instead of `::` to the test name, the command will fail.
The error will be `file or directory not found` and pytest will exit with error code 4 (which means "pytest command line usage error")
## Debug a Single Test (Inside The Tester Pod)

### 1) Prerequisites:
Following the 'Run a Single Ceph S3 Test' steps.
### 2) View The Test Content
You can view the test by going to the test file and searching for the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vi ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`.
You can view the test by going to the test file and searching for the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vim ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`.

The best place to start investigating is noobaa endpoint pod logs. if you are running with debug level that is higher than 1, you should see log messages of the S3 requests with the prefix `S3 REQUEST`. S3 replies will be with the prefix `HTTP REPLY`.

### 3) Change a Test
Sometimes you would like to change a test: add printing of variables, skip an assertion as needed, or you suspect that it has a faulty and you would like to change the code.

Since the file `./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` is a read-only file, decide on one of the options:

#### A. Permanent change - this change will be saved in a repo, it is for continues investigating.
1) Fork and clone the repository [ceph/s3-test](https://github.com/ceph/s3-tests).
2) Create a new branch from the hash number that was set in the file `./src/test/system_tests/ceph_s3_tests/test_ceph_s3_deploy.sh`.
Expand All @@ -206,7 +213,7 @@ Since the file `./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/fun
5) Build the tester image again, deploy noobaa, and run the test (repeat the steps starting from 'Build Core And Tester Images (Noobaa-Core)' above).

#### B. Temporary change - this change will be saved in the file inside the container, useful when you need a small change.
You can edit a test by going to the test file and editing the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vi ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`.
You can edit a test by going to the test file and editing the test function. e.g. if you are working on test `s3tests_boto3.functional.test_s3:test_set_bucket_tagging` then you should `vim ./src/test/system_tests/ceph_s3_tests/s3-tests/s3tests_boto3/functional/test_s3.py` and search for the function `test_set_bucket_tagging`.
## Compare to AWS Response (Inside Tester Pod)
Prerequisites:
Following the 'Run a Single Ceph S3 Test' steps until 'Deploy The Tester Deployment (Noobaa-Core Tab)'.
Expand All @@ -221,7 +228,6 @@ In this section we will do some manual changes that will allow you to check AWS
vim src/test/system_tests/ceph_s3_tests/test_ceph_s3_config_aws.conf
```
* host = s3.amazonaws.com
* bucket prefix = choose_name (for example: `bucket prefix = foo-bucket` you will need to manually delete it from AWS, and its name will be `foo-bucket1`, it adds suffix of 1).
* access_key, secret_key appears 3 times each in the file.
3) Running tests with the new configuration files will run against AWS:
```bash
Expand All @@ -235,15 +241,19 @@ S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config_aws.c
Following the 'Run All Ceph S3 Tests' steps.

### 1) All Running Tested Passed
a snippet from the last part of running all the tests.
a snippet from the last part of running all the tests, Before the list of skipped and failed tests.
You can see how many tests run and a status for each test.
```
...
Test Passed: s3tests_boto3.functional.test_sts.test_assume_role_with_web_identity_resource_tag_copy_obj
Test Passed: s3tests_boto3.functional.test_sts.test_assume_role_with_web_identity_role_resource_tag
Test Passed: s3tests_boto3.functional.test_utils.test_generate
Finished Running Ceph S3 Tests
CEPH TEST SUMMARY: Suite contains 812, ran 387 tests, Passed: 387, Skipped: 0, Failed: 0
Test Passed: s3tests_boto3/functional/test_s3select.py::test_bool_cast_expressions
Test Passed: s3tests_boto3/functional/test_s3select.py::test_output_serial_expressions
Test Passed: s3tests_boto3/functional/test_utils.py::test_generate
inished Running Ceph S3 Tests
CEPH TEST SUMMARY: Suite contains 779, ran 336 tests, Passed: 310, Skipped: 26, Failed: 0
CEPH TEST SKIPPED TESTS SUMMARY: 26 skipped tests
s3tests/functional/test_s3.py::test_object_storage_class
s3tests/functional/test_s3.py::test_object_storage_class_multipart
...
```
## Running a Single Test

Expand All @@ -259,7 +269,7 @@ S3TEST_CONF=${PWD}/src/test/system_tests/ceph_s3_tests/test_ceph_s3_config.conf

Note that there is the warning:
`WARNING: could not copy distfile to //.tox/distshare`
this warning is for tox to use the same dependancies between projects. this feature is depricated and not used on this project. in order to remove the warning you can modify src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini to include the following line in the `[tox]` section: `distshare = /root/node_modules/noobaa-core/.tox/distshare`
this warning is for tox to use the same dependancies between projects. this feature is deprecated and not used on this project. In order to remove the warning you can modify src/test/system_tests/ceph_s3_tests/s3-tests/tox.ini to include the following line in the `[tox]` section: `distshare = /root/node_modules/noobaa-core/.tox/distshare`

### 2) Test Fail
For example:
Expand Down
3 changes: 2 additions & 1 deletion docs/dev_guide/ceph_s3_tests/ceph_s3_tests_pending_list_status.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,5 @@ Attached a table with tests that where investigated and their status (this table
| test_schema_definition | Faulty Test | | Same as test_alias_cyclic_refernce |
| test_progress_expressions | Faulty Test | [508](https://github.com/ceph/s3-tests/issues/508) | |
| test_lifecycle_set_date | Faulty Test | [510](https://github.com/ceph/s3-tests/issues/510) | |
| test_lifecycle_transition_set_invalid_date | Not Implemented | | added because of the following PR [7270](https://github.com/noobaa/noobaa-core/pull/7270#discussion_r1175123422) |
| test_lifecycle_transition_set_invalid_date | Not Implemented | | added because of the following PR [7270](https://github.com/noobaa/noobaa-core/pull/7270#discussion_r1175123422) |
| All aws4 tests (e.g test_bucket_create_bad_amz_date_before_epoch_aws4) | Faulty Test | [520](https://github.com/ceph/s3-tests/issues/520) | |
39 changes: 1 addition & 38 deletions src/test/system_tests/ceph_s3_tests/s3-tests-lists/s3_tests_black_list.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -352,41 +352,4 @@ s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_resourc
s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_wrong_resource_tag_deny
s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_resource_tag_princ_tag
s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_resource_tag_copy_obj
s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_role_resource_tag
s3tests/functional/test_headers.py::test_object_create_bad_md5_invalid_garbage_aws4
s3tests/functional/test_headers.py::test_object_create_bad_contentlength_mismatch_below_aws4
s3tests/functional/test_headers.py::test_object_create_bad_authorization_incorrect_aws4
s3tests/functional/test_headers.py::test_object_create_bad_authorization_invalid_aws4
s3tests/functional/test_headers.py::test_object_create_bad_ua_empty_aws4
s3tests/functional/test_headers.py::test_object_create_bad_ua_none_aws4
s3tests/functional/test_headers.py::test_object_create_bad_date_invalid_aws4
s3tests/functional/test_headers.py::test_object_create_bad_amz_date_invalid_aws4
s3tests/functional/test_headers.py::test_object_create_bad_date_empty_aws4
s3tests/functional/test_headers.py::test_object_create_bad_amz_date_empty_aws4
s3tests/functional/test_headers.py::test_object_create_bad_date_none_aws4
s3tests/functional/test_headers.py::test_object_create_bad_amz_date_none_aws4
s3tests/functional/test_headers.py::test_object_create_bad_date_before_today_aws4
s3tests/functional/test_headers.py::test_object_create_bad_amz_date_before_today_aws4
s3tests/functional/test_headers.py::test_object_create_bad_date_after_today_aws4
s3tests/functional/test_headers.py::test_object_create_bad_amz_date_after_today_aws4
s3tests/functional/test_headers.py::test_object_create_bad_date_before_epoch_aws4
s3tests/functional/test_headers.py::test_object_create_bad_amz_date_before_epoch_aws4
s3tests/functional/test_headers.py::test_object_create_bad_date_after_end_aws4
s3tests/functional/test_headers.py::test_object_create_bad_amz_date_after_end_aws4
s3tests/functional/test_headers.py::test_object_create_missing_signed_custom_header_aws4
s3tests/functional/test_headers.py::test_object_create_missing_signed_header_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_authorization_invalid_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_ua_empty_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_ua_none_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_date_invalid_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_invalid_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_date_empty_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_empty_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_date_none_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_none_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_today_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_before_today_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_date_after_today_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_after_today_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_date_before_epoch_aws4
s3tests/functional/test_headers.py::test_bucket_create_bad_amz_date_before_epoch_aws4
s3tests_boto3/functional/test_sts.py::test_assume_role_with_web_identity_role_resource_tag
Loading

0 comments on commit b8e288a

Please sign in to comment.