Skip to content

feat: document jobconfig file #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 36 commits into from
Mar 28, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
6cc5c84
Add start:dev command
sbliven Oct 14, 2024
47d2467
Reformat backend/configuration.md
sbliven Oct 14, 2024
caf1894
Add jobconfig documentation
sbliven Oct 14, 2024
23ac019
Convert json job configuration to yaml
sbliven Oct 29, 2024
0da65eb
Update job documentation
sbliven Nov 3, 2024
1a9dc3f
document rabbitmq
despadam Jan 8, 2025
a3c2844
document job groups
despadam Jan 8, 2025
94c3df1
document job auth
despadam Jan 8, 2025
05baec0
document job tests - wip
despadam Jan 9, 2025
31acd6f
document job tests done
despadam Jan 9, 2025
941a2f4
fixes job test
despadam Jan 10, 2025
3d8b8dd
fix variable names in jobs tests
despadam Jan 10, 2025
82b9ac7
refactored job tests update
sofyalaski Jan 14, 2025
7d765bc
Update Development/v4.x/backend/testing/jobs.md
despadam Jan 16, 2025
32d163a
Merge pull request #1 from sofyalaski/refactor-job-tests
despadam Jan 16, 2025
5303c4e
document email and url actions
despadam Jan 24, 2025
cac610b
rename statusUpdate to update
despadam Feb 28, 2025
76a27cd
add test for anonymous user
despadam Mar 14, 2025
61da2ac
Merge pull request #2 from sbliven/fix-1758
despadam Mar 14, 2025
cb0f333
changes in backend #1774
sofyalaski Mar 20, 2025
c851a34
Merge branch 'master' into jobconfig
despadam Mar 20, 2025
68ee700
add a new last test
sofyalaski Mar 20, 2025
7f857d3
Merge branch 'jobconfig' into jobs-read-permission
despadam Mar 20, 2025
893cca8
Merge pull request #4 from sofyalaski/jobs-read-permission
despadam Mar 20, 2025
2c9322d
changes from release-jobs
sofyalaski Mar 21, 2025
2d17400
Update Development/v4.x/backend/testing/jobs.md
sofyalaski Mar 21, 2025
989780d
add changes after review
sofyalaski Mar 21, 2025
3afc578
changes after review
sofyalaski Mar 21, 2025
61b8fe3
Merge pull request #5 from sofyalaski/release-jobs-tests
despadam Mar 21, 2025
4a373e4
fix typos
despadam Mar 26, 2025
8003a17
fix typos in jobs tests
despadam Mar 26, 2025
624b421
fix last typos
despadam Mar 26, 2025
ab89e1d
add a note that admin can do any job
sofyalaski Mar 28, 2025
1516972
Merge pull request #6 from sofyalaski/admin-jobs
despadam Mar 28, 2025
a302b0d
fix config
despadam Mar 28, 2025
cee0ae6
finish config for jobs and emails
despadam Mar 28, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,7 @@ docs
*.pdf
.idea
cat

# Website output
_site
.jekyll-cache
2 changes: 1 addition & 1 deletion Development/v4.x/Data_Model.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ The data model and its API endpoints are described [here](https://scicatproject.

### Job

When a user wants to archive, retrieve or publish a dataset, SciCat must interact with the facility's storage and archiving system. This is done by creating a `Job` to track the interaction. The exact actions taken for each Job are often customized for each facility. A common action would be to forward the new Job entry to a Message Broker, e.g. RabbitMQ, from where it can be picked up by any program willing to react to this Job. For example, at PSI the RabbitMQ queue is emptied by a Node-RED process, which reads the Job information from the RabbitMQ queue. Alternatively, you can use other messaging solutions, such as Apache Kafka. In this way the (site specific) logic to handle the Jobs is kept outside the core of the SciCat system, giving a greater degree of flexibility. The external systems should ideally respond with status updates to the Job model, when there job is finished. As the job progresses, the system can make PUT/PATCH API calls back that update the `Job Status Message` and, when necessary, also the individual status of the involved datasets by updating the DatasetLifecycle information.
When a user wants to archive, retrieve or publish a dataset, SciCat must interact with the facility's storage and archiving system. This is done by creating a `Job` to track the interaction. The exact actions taken for each Job are often customized for each facility. A common action would be to forward the new Job entry to a Message Broker, e.g. RabbitMQ, from where it can be picked up by any program willing to react to this Job. For example, at PSI the RabbitMQ queue is emptied by a Node-RED process, which reads the Job information from the RabbitMQ queue. Alternatively, you can use other messaging solutions, such as Apache Kafka. In this way the (site specific) logic to handle the Jobs is kept outside the core of the SciCat system, giving a greater degree of flexibility. The external systems should ideally respond with updates to the Job model, when the job is finished. As the job progresses, the system can make PUT/PATCH API calls back that update the `Job Status` etc and, when necessary, also the individual status of the involved datasets by updating the DatasetLifecycle information.

The data model and its API endpoints are described [here](https://scicatproject.github.io/api/#operation/Job.create).

Expand Down
68 changes: 34 additions & 34 deletions Development/v4.x/backend/authorization/authorization_jobs.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
# Jobs Authorization
## CASL ability actions
This is the list of the permissions methods available for Jobs and all their endpoints.
The authorization for jobs is consistently different from all the other endpoints
This is the list of the permission methods available for Jobs and all their endpoints.
The authorization for jobs is consistently different from all the other endpoints.
Users in `ADMIN_GROUPS` have elevated access and are allowed to do any operation listed bellow.

### Endpoint Authorization
- JobCreate
- JobRead
- JobStatusUpdate
- JobUpdate
- JobDelete

### (Data) Instance Authorization
- JobCreateConfiguration (The jobs create section of the configuration dictats if the user can create the job)
- JobCreateOwner (Users with this privileges can create jobs only for themselves)
- JobCreateAny (Users with this privileges can create jobs for any of the users that are defined in the create section of the job configuration)
- JobCreateConfiguration (The job's create section of the configuration dictates if the user can create the job)
- JobCreateOwner (Users with this privilege can create jobs only for themselves)
- JobCreateAny (Users with this privilege can create jobs for any of the users that are defined in the create section of the job configuration)
- JobReadAccess
- JobReadAny
- JobStatusUpdateConfiguration (The jobs update section in configuration dictates if the user can update the status of the job)
- JobStatusUpdateOwner (Users with this privileges can update the status of jobs belonging to themselves)
- JobStatusUpdateAny (Users with this privileges can update the status of any job)
- JobDeleteAny
- JobUpdateConfiguration (The job's update section in configuration dictates if the user can update the job)
- JobUpdateOwner (Users with this privilege can update jobs belonging to themselves)
- JobUpdateAny (Users with this privilege can update any job)

#### Priority
```mermaid
Expand All @@ -27,53 +27,53 @@ graph LR;
JobCreateConfiguration-->JobCreateAny;
JobRead-->JobReadAccess;
JobReadAccess-->JobReadAny;
JobStatusUpdate-->JobStatusUpdateConfiguration;
JobStatusUpdateConfiguration-->JobStatusUpdateAny;
JobDelete-->JobDeleteAny;
JobUpdate-->JobUpdateConfiguration;
JobUpdateConfiguration-->JobUpdateAny;
JobDelete;
```

#### Authorization table
| HTTP method | Endpoint | Endpoint Authentication | Anonymous | Authenticated | Create Jobs Groups | Update Jobs Groups | Admin Groups | Delete Groups | Notes |
| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |
| POST | Jobs | _JobCreate_ | _JobCreateConfiguration_ | _JobCreateConfiguration_ | Any<br>_JobsCreateOwner_ | __no__ | Any<br>_JobsCreateAny_ | __no__ | |
| GET | Jobs | _JobReadMany_ | __no__ | Has Access<br>_JobReadAccess_ | Has Access<br>_JobReadAccess_ | __no__ | Any<br>_JobReadAny_ | __no__ | |
| GET | Jobs/_jid_ | _JobReadOne_ | __no__ | Has Access<br>_JobReadAccess_ | Has Access<br>_JobReadAccess_ | __no__ | Any<br>_JobReadAny_ | __no__ | |
| POST | Jobs/statusUpdate | _JobStatusUpdate_ | __no__ | _JobStatusUpdateConfiguration_ | __no__ | Owner<br>_JobStatusUpdateOwner_ | Any<br>_JobStatusUpdateAny_ | __no__ | |
| DELETE | Jobs/_jid_ | _JobDelete_ | __no__ | __no__ | __no__ | __no__ | __no__ | _JobDeleteAny_ | |
| HTTP method | Endpoint | Endpoint Authentication | Anonymous | Authenticated | Create Jobs Groups | Update Jobs Groups | Admin Groups | Delete Groups |
| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |
| POST | Jobs | _JobCreate_ | _JobCreateConfiguration_ | _JobCreateConfiguration_ | Any<br>_JobsCreateOwner_ | __no__ | Any<br>_JobsCreateAny_ | __no__ |
| GET | Jobs | _JobReadMany_ | __no__ | Has Access<br>_JobReadAccess_ | Has Access<br>_JobReadAccess_ | __no__ | Any<br>_JobReadAny_ | __no__ |
| GET | Jobs/_jid_ | _JobReadOne_ | __no__ | Has Access<br>_JobReadAccess_ | Has Access<br>_JobReadAccess_ | __no__ | Any<br>_JobReadAny_ | __no__ |
| PATCH | Jobs/_jid_ | _JobUpdate_ | __no__ | _JobUpdateConfiguration_ | __no__ | Owner<br>_JobUpdateOwner_ | Any<br>_JobUpdateAny_ | __no__ |
| DELETE | Jobs/_jid_ | _JobDelete_ | __no__ | __no__ | __no__ | __no__ | __no__ | __no__ |

#### Job Create Authorization Table
The _JobCreateConfiguration_ authorization permissions are configured directly in the __*create*__ section of the job configuration.
Any positive match will results in the user acquiring _JobCreate_ endpoint authorization which apply to the jobs endpoint POST:Jobs
Any positive match will result in the user acquiring _JobCreate_ endpoint authorization, which applies to the jobs endpoint `POST:Jobs`

| Job Create Authorization | Endpoint Authentication Translation | Endpoint Authentication Description | Instance Authentication Translation | Instance Authentication Description |
| --- | --- | --- | --- | --- |
| _#all_ | _#all_ | any user can access this endpoint, both anonymous and authenticated | _#all_ | Any user can create this instance of the job |
| _#datasetPublic_ | _#all_ | any user can access this endpoint, both anonymous and and authenticated | _#datasetPublic_ | the job instance will be created only if all the datasets listed are __public__ |
| _#authenticated_ | _#user_ | any valid users can access the endpoint, independently from their groups | _#user_ | any valid users can cretae this instance of the job |
| _#datasetPublic_ | _#all_ | any user can access this endpoint, both anonymous and authenticated | _#datasetPublic_ | the job instance will be created only if all the datasets listed are __public__ |
| _#authenticated_ | _#user_ | any valid users can access the endpoint, independently from their groups | _#user_ | any valid users can create this instance of the job |
| _#datasetAccess_ | _#user_ | any valid user can access this endpoint, independently from their groups | _#datasetAccess_ | the job instance will be created only if the user has access to all the datasets listed |
| _#datasetOwner_ | _#user_ | any valid user can access this endpoint, independently from their groups | _#datasetOwner_ | the job instance will be created only if the user is part of all the datasets owner group |
| _#datasetOwner_ | _#user_ | any valid user can access this endpoint, independently from their groups | _#datasetOwner_ | the job instance will be created only if the user is part of all the datasets' owner group |
| __*@GROUP*__ | __*GROUP*__ | only users that belongs to the specified group can access the endpoint | __*GROUP*__ | the job instance will be created only if the user belongs to the group specified |
| __*USER*__ | __*USER*__ | only the specified user can access the endpoint | __*USER*__ | the job instance can be created only by the user indicated |

__IMPORTANT__: use option _#all_ carefully, as it allows anybody to create a new job. It is mostly use for debuging and testing
__IMPORTANT__: use option _#all_ carefully, as it allows anybody to create a new job. It is mostly used for debuging and testing.

#### Job Status Update Authorization Table
The _JobStatusUpdateConfiguration_ authorization permissions are configured directly in the __*update*__ section of the job configuration.
Any positive match will results in the user acquiring _JobStatusUpdate_ endpoint authorization apply to the jobs endpoint POST:Jobs/statusUpdate
#### Job Update Authorization Table
The _JobUpdateConfiguration_ authorization permissions are configured directly in the __*update*__ section of the job configuration.
Any positive match will result in the user acquiring _JobUpdate_ endpoint authorization, which applies to the jobs endpoint `PATCH:Jobs/id`

| Job Status Update Authorization | Endpoint Authentication Translation | Endpoint Authentication Description | Instance Authentication Translation | Instance Authentication Description |
| Job Update Authorization | Endpoint Authentication Translation | Endpoint Authentication Description | Instance Authentication Translation | Instance Authentication Description |
| --- | --- | --- | --- | --- |
| _#all_ | _#all_ | any user can access this endpoint, both anonymous and authenticated | _#all_ | Any user can update the status of this job instance |
| _#all_ | _#all_ | any user can access this endpoint, both anonymous and authenticated | _#all_ | Any user can update this job instance |
| _#jobOwnerUser_ | _#user_ | authenticated user can access the endpoint | _#jobOwnerUser_ | only the user that is listed in field _ownerUser_ can perform the update |
| _#jobOwnerGroup_ | _#user_ | authenticated user can access the endpoint | _#jobOwnerGroup_ | any user that belongs to the group listed in field _ownerGroup_ can perform the update |
| __*@GROUP*__ | __*GROUP*__ | only users that belongs to the specified group can access the endpoint | __*GROUP*__ | the job status can be updated only by users who belong to the group specified |
| __*USER*__ | __*USER*__ | only the specified user can access the endpoint | __*USER*__ | the job status can be updated only by the user indicated |
| __*@GROUP*__ | __*GROUP*__ | only users that belong to the specified group can access the endpoint | __*GROUP*__ | the job can be updated only by users who belong to the group specified |
| __*USER*__ | __*USER*__ | only the specified user can access the endpoint | __*USER*__ | the job can be updated only by the user indicated |

__IMPORTANT__: use option _#all_ carefully, as it allows anybody to update the status of the job. It is mostly use for debuging and testing
__IMPORTANT__: use option _#all_ carefully, as it allows anybody to update the job. It is mostly used for debuging and testing.

#### Job Authorization priority
The endpoint authorization is the most permissive authorization across all the jobs defined.
The priority between job create and status update authorization is as follow:
The priority between job create and update authorization is as follows:

```mermaid
graph LR;
Expand Down
Loading