Mc issue 310 refactor security #320
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
An exception handling decorator named handle_exceptions was introduced to simplify the handling of exceptions in most major resources. The decorator eliminates the need to manually try/except blocks in each route, thus reducing repetitive code and enhancing readability. All routes in resources like User.py, ResetTokenValidation.py, RequestResetPassword.py, and more were updated to use this decorator.
This commit adds a new python file containing unit tests for middleware functions that interact with the database. These tests include checking for the addition, deletion, retrieval, and update of various data sources, users, and reset tokens to ensure the correct operation and accuracy of these functions.
This commit introduces a new helper file for testing middleware interaction with the database. It includes test setup methods for generating fake data in the database, such as users, data sources, and reset tokens, which will provide a solid basis for thorough testing of database-related middleware functionality.
This commit both minimizes test implementation detail within individual tests and also enhances testing utilities. Specifically, large helper functions for creating test users and reset tokens have been moved from the test file into a separate helper file. Additionally, a new comprehensive test for the 'quick_search_query' function has been introduced to ensure proper functionality.
Renamed 'helper_test_middleware.py' to 'middleware/helper_functions.py', and made modifications to test data inserts. Added new testing helper functions for quick search query logs and boolean dictionary creation. This commit includes renaming a test file along with additions and modifications to the test data insert statements. New test helper functions have been introduced to handle the quick search query logs and create a dictionary of booleans. This refactor aims at improving the test coverage and overall project organization.
Added two testing fixtures for middleware tests in a new `fixtures.py` file within tests/middleware directory. The fixtures provide functionality for setting up a connection to the development database and creating a cursor for database operations. They also ensure that changes made during tests can be rolled back using PostgreSQL's SAVEPOINT management. This is done to prevent any permanent changes to the database resulting from the testing process.
Added several new tests for data source query functions in a new file, `test_data_source_queries.py`. These tests cover different aspects like getting approved data sources, data source by id and data sources for map. Additional setup and teardown functionality were also added through two fixtures, 'inserted_data_sources_found' and 'connection_with_test_data' for necessary test setup and cleanup processes.
Moved the 'connection_with_test_data' fixture from 'test_data_source_queries.py' to 'fixtures.py' to improve organization and reusability..
This commit introduces two new tests for the archives_get_results and archives_get_columns methods in the middleware layer. These tests validate the method returns and interaction with the database, enhancing the overall test coverage.
This commit introduces tests for the login_queries middleware, specifically for the login_results, create_session_token, and is_admin methods. These new tests ensure these methods interact properly with the database and improve the test coverage in our middleware layer.
Added optional return type and function description for get_most_recent_quick_search_query_log in tests/middleware/helper_functions.py. Also, handling no-result situation by checking if the result is None. These enhancements improve function's readability and robustness.
Implemented unit tests for the quick search query methods, specifically `unaltered_search_query` and `quick_search_query`. These tests ensure that search query results and logging function properly and return expected outputs. This includes checks on the length of responses, expected keys, and datetime accuracy in logs.
Implemented unit tests for user and reset token query methods in the middleware. These tests validate the functionality of creating, checking, and deleting users and reset tokens, ensuring the correct interaction with the database. The tests also validate the return values and the changes made to the database state.
The README has been updated to reflect changes in the project's testing structure. The new instructions include details on where unit and integration tests for the API are located, and the need to set up an environment variable for tests requiring database access. The instructions on how to run the tests, using the `pytest` command, and their interaction with CI/CD pipelines, remain unchanged.
The Python test workflow file has been renamed from python_tests.yml to test_api.yml and fully configured to run tests using GitHub Actions. It's been set up to specifically test the project's API on pull requests with environment variables for secrets and the pytest command to run the tests. It also upgrades pip and downloads dependencies.
The test imports in test_data_source_queries.py have been updated. This revision includes modifying the import path for app_test_data and removing a redundant import from middleware.fixtures.
The test files have been relocated to tests/resources directory and their respective import paths have been updated accordingly. Additionally, unused imports have been removed from the middleware tests, thus enhancing code readability and maintenance.
Optimized code for better readability through improving docstrings and line break formatting. Added flake8 argument to ignore E501 (line too long) to avoid warnings related to line exceeding maximum length.
Expanded the ignored warnings in flake8 configuration to include trailing whitespaces, increasing flexibility in code style. Simultaneously, improved clarity and readability of test modules by refining docstrings and line alignment.
Expanded the ignored warnings in flake8 configuration to include trailing whitespaces, increasing flexibility in code style. Simultaneously, improved clarity and readability of test modules by refining docstrings and line alignment.
A custom exception, DatabaseInitializationError, has been added for scenarios where psycopg2 connection initialization fails. The exception handling in the `initialize_psycopg2_connection` function and usage in other places has been adapted accordingly. Furthermore, skeleton tests for successful and unsuccessful psycopg2 connection initialization have been added.
Updated the middleware to handle the UserNotFoundError and implemented the custom exception for when a user is not found in the login_queries script. Also refined the return type of is_admin function to only boolean, managing the missing user situation with the new custom exception.
The code has been refactored to individual query methods for clear data handling and improve readability. The 'data_sources_query' method in 'DataSources.py' which was previously handling all query types is replaced with specific, self-explanatory query methods such as 'get_approved_data_sources', 'needs_identification_data_sources' and 'get_data_sources_for_map'. This also leads to the introduction of 'convert_data_source_matches' in 'data_source_queries.py' file for better format conversion.
The archives_put_query method has been removed from the archives_queries.py and its functionality has been split into direct calls to the appropriate methods, archives_put_broken_as_of_results or archives_put_last_cached_results. This change simplifies the code and makes it more legible by clearly specifying the function being used in each situation. The refactoring also involves adjustments in the Archives.py resource to accommodate these changes.
In the helper function and tests module, the SQL queries were updated to use `DATETIME_OF_REQUEST` instead of `UPDATED_AT`. In `test_quick_search_query.py`, the code to get the current date was changed to pull directly from the database using 'SELECT NOW()'. Removed redundancy in data_source_by_id_query assertions and simplified datetime comparison in quick_search_query test. Made necessary import adjustments in `test_data_source_queries.py`.
ℹ️ Added test_RefreshSession.py file with test cases for refreshing session functionality, including happy path, token not found, and unexpected error scenarios.
…e_308_fix_Refresh_Session_bug Mc issue 308 fix refresh session bug
…me field - Removed the redundant 'datetime_of_request' field from the INSERT_LOG_QUERY in the quick_search_query.py file in the middleware directory - This fixes an issue where unnecessary data was being inserted into the quick_search_query_logs table - The query now only includes essential fields for logging search queries and results
- Removed redundant psycopg2_connection initialization and query duplication to improve code readability and efficiency.
…e_290_remove_datetime_of_request Mc issue 290 remove datetime of request
…285_refactor_search_tokens # Conflicts: # app.py # resources/DataSources.py # resources/QuickSearch.py # resources/SearchTokens.py # tests/test_endpoints.py
…se status code and add missing source_url key check in test_data_sources_by_id.py
…e_285_refactor_search_tokens Mc issue 285 refactor search tokens
Refactored the function signatures in quick_search_query.py to improve readability and maintainability. Updated the function quick_search_query_wrapper to include type hints for arguments and return value. Made adjustments to the INSERT_LOG_QUERY format function call for consistency.
ℹ️ Enhance data_source_by_id_wrapper function to explicitly return a Response object for clarity and consistency.
ℹ️ Added tests for the data_source_by_id_wrapper function to cover scenarios when data is found and when data is not found.
…redundant tests of associated Resource Class
…e_290_remove_datetime_of_request Rearrange and add Quick Search Query and DataSources tests
ℹ️ Fix the status code returned when a data source is not found to be 200 instead of 404 in middleware/data_source_queries.py and tests/middleware/test_data_source_queries.py.
…e_312_replace_http_status_code_when_source_not_found ⚙️ Adjust status code for data source not found response
…310_refactor_security # Conflicts: # tests/helper_functions.py
…dation and role checking.
Removed redundant NoAPIKeyError handling in security middleware to improve code readability and maintainability.🛠️
|
Whoops! Wrong branch! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes
security.pyand add error message for insufficient permissions #310Description
is_validandapi_requiredto have more straightforward logic, utilize function extraction where possible, and replace magic numbers and magic strings with reusable constants and functions.middleware\test_security.py, providing 100% coverage.Testing
Performance
Docs