Mc data requests endpoint #4
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A new Python class named 'DataRequestsManager' has been added in the 'data_requests.py' file in middleware. This class is responsible for managing CRUD operations (create, read, update, delete) for the 'data_requests' table in a PostgreSQL database. The class uses psycopg2 for database interaction.
An exception handling wrapper function 'handle_exceptions' has been added to the 'PsycopgResource.py'. It wraps all method calls in a try-except block that rolls back database transactions immediately upon a failure and returns an error message.
A new file 'DataRequest.py' has been added to handle CRUD operations on 'data_requests' table. Specifically, the added 'DataRequest' class extends 'PsycopgResource' and performs create, read, update, and delete operations through the use of methods like 'post', 'get', 'put', and 'delete', all of which implement additional security and exception handling layers.
Created a `StarredDataSourceManager` class for CRUD operations on the 'user_starred_data_sources' table in a PostgreSQL database. Also introduced a Flask-Restful resource `UserStarDatasource` for handling request methods related to user-starred data sources. Changes have been incorporated in the app's routing.
This reverts commit cf3e677.
This reverts commit 03093e5.
The get, put, and delete methods in DataRequest.py have been altered to extract the 'request_id' from the request body. Previously, these methods required the 'request_id' as an argument. The changes ensure that the 'request_id' is always available before performing associated actions, improving the flexibility and efficiency of our API request handling.
The DataRequest resource is now imported and one of its routes is added to the application's URL structure. This will enable us to access DataRequest actions through HTTP requests, expanding our API functionality.
This commit adds a new python file containing unit tests for middleware functions that interact with the database. These tests include checking for the addition, deletion, retrieval, and update of various data sources, users, and reset tokens to ensure the correct operation and accuracy of these functions.
This commit introduces a new helper file for testing middleware interaction with the database. It includes test setup methods for generating fake data in the database, such as users, data sources, and reset tokens, which will provide a solid basis for thorough testing of database-related middleware functionality.
This commit both minimizes test implementation detail within individual tests and also enhances testing utilities. Specifically, large helper functions for creating test users and reset tokens have been moved from the test file into a separate helper file. Additionally, a new comprehensive test for the 'quick_search_query' function has been introduced to ensure proper functionality.
Renamed 'helper_test_middleware.py' to 'middleware/helper_functions.py', and made modifications to test data inserts. Added new testing helper functions for quick search query logs and boolean dictionary creation. This commit includes renaming a test file along with additions and modifications to the test data insert statements. New test helper functions have been introduced to handle the quick search query logs and create a dictionary of booleans. This refactor aims at improving the test coverage and overall project organization.
Added two testing fixtures for middleware tests in a new `fixtures.py` file within tests/middleware directory. The fixtures provide functionality for setting up a connection to the development database and creating a cursor for database operations. They also ensure that changes made during tests can be rolled back using PostgreSQL's SAVEPOINT management. This is done to prevent any permanent changes to the database resulting from the testing process.
Added several new tests for data source query functions in a new file, `test_data_source_queries.py`. These tests cover different aspects like getting approved data sources, data source by id and data sources for map. Additional setup and teardown functionality were also added through two fixtures, 'inserted_data_sources_found' and 'connection_with_test_data' for necessary test setup and cleanup processes.
Moved the 'connection_with_test_data' fixture from 'test_data_source_queries.py' to 'fixtures.py' to improve organization and reusability..
This commit introduces two new tests for the archives_get_results and archives_get_columns methods in the middleware layer. These tests validate the method returns and interaction with the database, enhancing the overall test coverage.
This commit introduces tests for the login_queries middleware, specifically for the login_results, create_session_token, and is_admin methods. These new tests ensure these methods interact properly with the database and improve the test coverage in our middleware layer.
Added optional return type and function description for get_most_recent_quick_search_query_log in tests/middleware/helper_functions.py. Also, handling no-result situation by checking if the result is None. These enhancements improve function's readability and robustness.
Implemented unit tests for the quick search query methods, specifically `unaltered_search_query` and `quick_search_query`. These tests ensure that search query results and logging function properly and return expected outputs. This includes checks on the length of responses, expected keys, and datetime accuracy in logs.
Implemented unit tests for user and reset token query methods in the middleware. These tests validate the functionality of creating, checking, and deleting users and reset tokens, ensuring the correct interaction with the database. The tests also validate the return values and the changes made to the database state.
The README has been updated to reflect changes in the project's testing structure. The new instructions include details on where unit and integration tests for the API are located, and the need to set up an environment variable for tests requiring database access. The instructions on how to run the tests, using the `pytest` command, and their interaction with CI/CD pipelines, remain unchanged.
The Python test workflow file has been renamed from python_tests.yml to test_api.yml and fully configured to run tests using GitHub Actions. It's been set up to specifically test the project's API on pull requests with environment variables for secrets and the pytest command to run the tests. It also upgrades pip and downloads dependencies.
The test imports in test_data_source_queries.py have been updated. This revision includes modifying the import path for app_test_data and removing a redundant import from middleware.fixtures.
The test files have been relocated to tests/resources directory and their respective import paths have been updated accordingly. Additionally, unused imports have been removed from the middleware tests, thus enhancing code readability and maintenance.
Optimized code for better readability through improving docstrings and line break formatting. Added flake8 argument to ignore E501 (line too long) to avoid warnings related to line exceeding maximum length.
Expanded the ignored warnings in flake8 configuration to include trailing whitespaces, increasing flexibility in code style. Simultaneously, improved clarity and readability of test modules by refining docstrings and line alignment.
Expanded the ignored warnings in flake8 configuration to include trailing whitespaces, increasing flexibility in code style. Simultaneously, improved clarity and readability of test modules by refining docstrings and line alignment.
In the helper function and tests module, the SQL queries were updated to use `DATETIME_OF_REQUEST` instead of `UPDATED_AT`. In `test_quick_search_query.py`, the code to get the current date was changed to pull directly from the database using 'SELECT NOW()'. Removed redundancy in data_source_by_id_query assertions and simplified datetime comparison in quick_search_query test. Made necessary import adjustments in `test_data_source_queries.py`.
The pytest command used in the .github/workflows/pull.yaml has been updated to correctly point to the app_test.py file. This adjustment affects the "Test with pytest" segment of the workflow.
Introduced two new enums, `RequestStatus` and `RecordType`, to parallel PostgreSQL enums in the database, and configured tests to ensure synchronization between Python and Database enums. This enables type safety and improves code readability.
Refactored DataRequestsManager to use the new DBRequestMapper class for generating database queries. Created new data classes RequestInfo and UpdateableRequestColumns to map request data to the database. Added appropriate tests to verify correct operation.
Refactored DataRequestsManager to use the new DBRequestMapper class for generating database queries. Created new data classes RequestInfo and UpdateableRequestColumns to map request data to the database. Added appropriate tests to verify correct operation.
This commit creates a new conftest.py file and adds pytest fixtures, one for creating a test client and another for establishing a database session. These fixtures help in setting up and tearing down the environment for each test. The test client is configured to connect to a pre-existing test database, and the session fixture is responsible for managing database sessions during testing, ensuring any changes made are rolled back after each test.
Added the SQLAlchemy dependency to the middleware models.py file. This change allows the use of SQLAlchemy's features, such as support for SQL queries and transactions, within our middleware.
Refactored data request functionality to use SQLAlchemy instead of psycopg2. This change simplifies CRUD operations in the 'data_requests' and 'requests_v2' tables, and reduces the amount of code needed to achieve these operations with SQLAlchemy's ORM features.
The data request handling in the application has been refactored to use SQLAlchemy instead of psycopg2. This shift leverages SQLAlchemy's ORM features for simpler and more efficient CRUD operations on the 'data_requests' and 'requests_v2' tables.
The test suite in middleware for data requests has been overhauled. It now integrates a new setup for mock test data, retiring previous testing methods. This change also includes the use of a new test client and the adoption of SQLAlchemy's ORM features for testing database interactions.
…ng_to_dev Mc testing to dev
The pytest fixtures in conftest.py were refactored for better database connection handling. A 'session' fixture now creates a scoped session, which is used to overwrite the existing 'db.session'. This setup ensures that changes are safely rolled back after each test, preventing potential test interferences. Additional imports related to SQLAlchemy were added.
Extended the middleware models.py to include SQLAlchemy models for Users and Requests V2 classes. Added new enum types: 'request_status_enum', and 'record_type_enum' to accommodate different status types and record types.
Added utility function to generate secure random alphanumeric strings. Also added functions to create test instances of User and RequestV2 models using SQLAlchemy. These changes will help in thoroughly testing the functionality and security of these models.
Replaced the usage of RequestInfo with the new RequestV2 model in the DataRequest class. This includes changes to the way instances of these classes are created, queried and transformed to dictionary. This refactoring aligns with the introduction of the new RequestV2 model and makes the DataRequest class compatible with it.
The code has been updated to use the RequestV2 model instead of the old RequestInfo class in test_data_requests.py. This entails changes to how instances are created, queried, and transformed into dictionaries. These modifications are in line with the introduction of the new RequestV2 model, ensuring compatibility across the project.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes
Description
Testing
Performance
Docs